Oct 30 17:58:15.317042 ip-10-0-141-106 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory Oct 30 17:58:15.317056 ip-10-0-141-106 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory Oct 30 17:58:15.317066 ip-10-0-141-106 systemd[1]: kubelet.service: Failed with result 'resources'. Oct 30 17:58:15.317395 ip-10-0-141-106 systemd[1]: Failed to start Kubernetes Kubelet. Oct 30 17:58:25.566341 ip-10-0-141-106 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found. Oct 30 17:58:25.566358 ip-10-0-141-106 systemd[1]: kubelet.service: Failed with result 'resources'. -- Boot 7b13398f400a49ec9a050ad3464b240b -- Oct 30 18:00:27.433067 ip-10-0-141-106 systemd[1]: Starting Kubernetes Kubelet... Oct 30 18:00:27.817351 ip-10-0-141-106 kubenswrapper[2577]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:00:27.817351 ip-10-0-141-106 kubenswrapper[2577]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 18:00:27.817351 ip-10-0-141-106 kubenswrapper[2577]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:00:27.817351 ip-10-0-141-106 kubenswrapper[2577]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:00:27.820639 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820563 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:27.820639 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820641 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820645 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820649 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820652 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820654 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820657 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820660 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820662 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820665 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820668 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820670 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820673 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820675 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820678 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820680 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820683 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820685 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820688 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820691 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820699 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:27.820707 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820701 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820704 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820706 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820710 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820712 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820715 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820718 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820722 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820724 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820727 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820729 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820732 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820734 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820737 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820740 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820742 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820745 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820747 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820750 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820752 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:27.821197 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820755 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820757 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820760 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820762 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820765 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820767 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820770 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820772 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820774 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820778 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820780 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820782 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820786 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820789 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820792 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820794 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820797 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820799 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820802 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:27.821676 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820806 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820810 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820814 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820816 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820819 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820821 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820824 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820826 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820829 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820831 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820834 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820837 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820840 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820843 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820845 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820848 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820850 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820853 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820855 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820858 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:27.822149 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820860 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820863 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820865 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820868 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820870 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820874 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820876 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820879 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820882 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820884 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820887 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820889 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820892 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820894 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820897 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820900 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820902 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820905 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820907 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:27.822632 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820910 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820913 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820915 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820920 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820922 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820925 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.820927 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821697 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821704 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821707 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821710 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821713 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821715 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821718 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821721 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821724 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821726 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821729 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821732 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821735 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:27.823075 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821737 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821740 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821743 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821745 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821748 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821750 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821753 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821755 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821758 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821760 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821763 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821765 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821768 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821770 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821773 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821776 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821778 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821780 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821783 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:27.823576 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821786 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821789 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821792 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821794 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821797 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821799 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821802 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821804 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821807 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821810 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821812 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821815 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821817 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821820 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821822 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821825 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821827 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821830 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821834 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:27.824026 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821838 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821841 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821843 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821846 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821849 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821852 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821854 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821856 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821859 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821861 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821864 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821866 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821869 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821871 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821874 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821877 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821879 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821882 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821884 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821887 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:27.824522 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821891 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821893 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821896 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821899 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821901 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821904 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821906 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821908 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821911 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821913 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821916 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821918 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821920 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821923 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821925 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821928 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821931 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821933 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821935 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821938 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:27.824982 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821940 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821943 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821945 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821948 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821950 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821953 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821955 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821959 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821962 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821965 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821967 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821970 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821972 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821975 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.821980 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822460 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822465 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822468 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822471 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:27.825461 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822474 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822477 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822479 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822481 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822484 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822487 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822489 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822492 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822494 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822497 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822499 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822502 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822504 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822507 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822509 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822512 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822514 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822517 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822519 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:27.825909 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822521 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822525 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822527 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822530 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822533 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822535 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822538 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822541 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822544 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822546 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822549 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822552 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822554 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822557 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822560 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822562 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822564 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822567 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822569 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822572 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:27.826348 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822574 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822577 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822579 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822582 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822584 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822587 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822589 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822591 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822594 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822596 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822599 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822601 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822603 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822606 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822609 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822611 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822614 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822616 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822619 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822622 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:27.826831 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822625 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822628 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822630 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822633 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822636 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822639 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822641 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822644 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822647 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822649 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822652 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822654 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822657 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822659 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822662 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822664 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822666 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822669 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822671 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822674 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:27.827285 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822676 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822678 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822681 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822684 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822686 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822688 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822691 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822695 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822699 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822702 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822706 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822709 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822712 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822714 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822717 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822719 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822722 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822724 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822727 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:27.827765 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822729 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822732 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822736 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.822740 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823672 2577 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823683 2577 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823691 2577 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823695 2577 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823700 2577 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823703 2577 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823707 2577 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823711 2577 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823715 2577 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823718 2577 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823721 2577 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823724 2577 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823727 2577 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823730 2577 flags.go:64] FLAG: --cgroup-root="" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823733 2577 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823736 2577 flags.go:64] FLAG: --client-ca-file="" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823739 2577 flags.go:64] FLAG: --cloud-provider="external" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823742 2577 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823746 2577 flags.go:64] FLAG: --cluster-domain="" Oct 30 18:00:27.828210 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823752 2577 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823755 2577 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823758 2577 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823762 2577 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823766 2577 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823769 2577 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823772 2577 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823776 2577 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823779 2577 flags.go:64] FLAG: --contention-profiling="false" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823782 2577 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823785 2577 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823788 2577 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823791 2577 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823795 2577 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823798 2577 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823801 2577 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823804 2577 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823807 2577 flags.go:64] FLAG: --enable-server="true" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823810 2577 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823815 2577 flags.go:64] FLAG: --event-burst="100" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823818 2577 flags.go:64] FLAG: --event-qps="50" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823821 2577 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823824 2577 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823827 2577 flags.go:64] FLAG: --eviction-hard="" Oct 30 18:00:27.828760 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823831 2577 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823834 2577 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823837 2577 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823840 2577 flags.go:64] FLAG: --eviction-soft="" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823843 2577 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823846 2577 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823849 2577 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823852 2577 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823857 2577 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823860 2577 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823863 2577 flags.go:64] FLAG: --feature-gates="" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823867 2577 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823870 2577 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823873 2577 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823877 2577 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823880 2577 flags.go:64] FLAG: --healthz-port="10248" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823883 2577 flags.go:64] FLAG: --help="false" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823886 2577 flags.go:64] FLAG: --hostname-override="ip-10-0-141-106.ec2.internal" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823889 2577 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823892 2577 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823895 2577 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823898 2577 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823901 2577 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823904 2577 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 18:00:27.829365 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823907 2577 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823910 2577 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823913 2577 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823916 2577 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823919 2577 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823922 2577 flags.go:64] FLAG: --kube-reserved="" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823925 2577 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823928 2577 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823931 2577 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823934 2577 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823937 2577 flags.go:64] FLAG: --lock-file="" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823939 2577 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823942 2577 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823945 2577 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823952 2577 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823955 2577 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823960 2577 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823962 2577 flags.go:64] FLAG: --logging-format="text" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823966 2577 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823969 2577 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823972 2577 flags.go:64] FLAG: --manifest-url="" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823974 2577 flags.go:64] FLAG: --manifest-url-header="" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823979 2577 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823982 2577 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823985 2577 flags.go:64] FLAG: --max-pods="110" Oct 30 18:00:27.829949 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823988 2577 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823991 2577 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823994 2577 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.823997 2577 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824000 2577 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824003 2577 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824006 2577 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824012 2577 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824015 2577 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824018 2577 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824021 2577 flags.go:64] FLAG: --pod-cidr="" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824024 2577 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824026 2577 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824029 2577 flags.go:64] FLAG: --pods-per-core="0" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824032 2577 flags.go:64] FLAG: --port="10250" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824036 2577 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824039 2577 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09e88f591e009548a" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824042 2577 flags.go:64] FLAG: --qos-reserved="" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824045 2577 flags.go:64] FLAG: --read-only-port="10255" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824048 2577 flags.go:64] FLAG: --register-node="true" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824050 2577 flags.go:64] FLAG: --register-with-taints="" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824055 2577 flags.go:64] FLAG: --registry-burst="10" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824059 2577 flags.go:64] FLAG: --registry-qps="5" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824062 2577 flags.go:64] FLAG: --reserved-cpus="" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824066 2577 flags.go:64] FLAG: --reserved-memory="" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824070 2577 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 18:00:27.830900 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824074 2577 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824076 2577 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824079 2577 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824082 2577 flags.go:64] FLAG: --runonce="false" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824085 2577 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824088 2577 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824091 2577 flags.go:64] FLAG: --seccomp-default="false" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824094 2577 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824096 2577 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824100 2577 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824103 2577 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824106 2577 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824109 2577 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824112 2577 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824115 2577 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824118 2577 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824121 2577 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824123 2577 flags.go:64] FLAG: --system-cgroups="" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824126 2577 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824132 2577 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824134 2577 flags.go:64] FLAG: --tls-cert-file="" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824137 2577 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824141 2577 flags.go:64] FLAG: --tls-min-version="" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824144 2577 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824147 2577 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 18:00:27.832142 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824150 2577 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824152 2577 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824156 2577 flags.go:64] FLAG: --v="2" Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824160 2577 flags.go:64] FLAG: --version="false" Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824165 2577 flags.go:64] FLAG: --vmodule="" Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824169 2577 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824173 2577 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824308 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824312 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824315 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824318 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824321 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824323 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824326 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824328 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824331 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824333 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824336 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824339 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824341 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824343 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:27.833214 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824346 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824348 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824351 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824355 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824359 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824362 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824364 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824367 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824369 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824372 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824374 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824377 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824379 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824383 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824386 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824390 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824393 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824395 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824398 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:27.833731 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824401 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824404 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824406 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824408 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824411 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824427 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824430 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824433 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824435 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824438 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824440 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824443 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824446 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824448 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824451 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824453 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824456 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824458 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824461 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:27.834294 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824464 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824466 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824469 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824471 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824474 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824476 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824478 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824482 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824485 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824489 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824492 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824494 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824497 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824501 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824503 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824506 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824508 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824511 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824513 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824516 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:27.835170 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824518 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824521 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824523 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824527 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824531 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824533 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824536 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824539 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824541 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824544 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824546 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824549 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824551 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824554 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824556 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824559 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824561 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824564 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824566 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:27.836077 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824570 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:27.836776 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824572 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:27.836776 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824576 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:27.836776 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824579 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:27.836776 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824581 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:27.836776 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824583 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:27.836776 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824586 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:27.836776 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824589 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:27.836776 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824591 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:27.836776 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824594 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:27.836776 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824597 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:27.836776 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824599 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:27.836776 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824602 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:27.836776 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824604 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:27.836776 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824607 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:27.836776 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.824634 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:00:27.836776 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824675 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824679 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824682 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824685 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824687 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824690 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824693 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824695 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824698 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824700 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824703 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824705 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824707 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824710 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824712 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824715 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824718 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824722 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824726 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824728 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:27.837300 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824731 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824733 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824735 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824738 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824741 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824744 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824747 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824749 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824751 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824754 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824757 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824759 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824763 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824766 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824769 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824772 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824775 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824777 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824779 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:27.837787 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824782 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824784 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824787 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824790 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824792 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824794 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824797 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824799 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824802 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824805 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824808 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824812 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824815 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824817 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824819 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824822 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824824 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824827 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824830 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:27.838353 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824833 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824835 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824838 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824840 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824843 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824845 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824848 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824850 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824853 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824855 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824858 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824860 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824863 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824865 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824867 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824870 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824872 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824875 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824877 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:27.839140 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824879 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824882 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824884 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824887 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824890 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824894 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824896 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824899 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824901 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824904 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824906 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824909 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824911 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824915 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824919 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824922 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824925 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824928 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824930 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824933 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:27.839734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824935 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824938 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824941 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824943 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824946 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824948 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824951 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824953 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.824956 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.833159 2577 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.833179 2577 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.834058 2577 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.834069 2577 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834177 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834186 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834192 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834197 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834202 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834207 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834212 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:27.840329 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834217 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834223 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834229 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834234 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834238 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834242 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834247 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834251 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834255 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834260 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834264 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834268 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834273 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834277 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834281 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834285 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834289 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834293 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834297 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:27.840824 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834301 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834305 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834310 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834323 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834329 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834333 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834337 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834342 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834347 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834351 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834355 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834359 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834363 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834367 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834371 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834376 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834380 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834384 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834388 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:27.841321 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834393 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834398 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834402 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834406 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834410 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834435 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834440 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834444 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834448 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834452 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834456 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834460 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834464 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834468 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834472 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834476 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834480 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834492 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834497 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834501 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:27.841802 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834506 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834509 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834513 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834517 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834521 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834526 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834530 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834534 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834538 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834542 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834547 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834551 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834555 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834560 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834564 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834568 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834572 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834577 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834581 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:27.842350 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834585 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834589 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834593 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834605 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834610 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834614 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834618 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834625 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834631 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834635 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834640 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834654 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834658 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834662 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834666 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834670 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834675 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834679 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834683 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:27.842941 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834687 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:27.844107 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834691 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:27.844107 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.834695 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:27.844107 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.834739 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:00:27.844107 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835025 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:27.844107 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835035 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:27.844107 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835040 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:27.844107 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835046 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:27.844107 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835050 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:27.844107 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835056 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:27.844107 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835061 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:27.844107 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835067 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:27.844107 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835071 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:27.844107 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835076 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:27.844107 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835081 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:27.844107 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835085 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:27.844107 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835089 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835094 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835099 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835104 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835109 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835113 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835120 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835125 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835129 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835143 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835147 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835151 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835156 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835160 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835164 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835168 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835172 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835176 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835180 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:27.844599 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835184 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835188 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835192 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835196 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835200 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835205 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835209 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835213 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835217 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835222 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835226 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835230 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835234 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835238 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835242 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835246 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835251 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835255 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835259 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835263 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:27.845212 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835268 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835272 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835276 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835287 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835292 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835296 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835300 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835304 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835309 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835313 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835317 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835321 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835325 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835329 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835335 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835339 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835343 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835347 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835351 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835355 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:27.845940 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835359 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835363 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835368 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835372 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835376 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835380 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835384 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835390 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835396 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835401 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835431 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835436 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835441 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835445 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835449 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835453 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835467 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835471 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835476 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:27.846630 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835479 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:27.847194 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835484 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:27.847194 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835488 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:27.847194 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835492 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:27.847194 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835496 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:27.847194 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835500 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:27.847194 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835504 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:27.847194 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835508 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:27.847194 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835512 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:27.847194 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835516 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:27.847194 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835520 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:27.847194 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835525 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:27.847194 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835529 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:27.847194 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835532 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:27.847194 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835537 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:27.847194 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835542 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:27.847194 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.835582 2577 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835644 2577 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835651 2577 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835655 2577 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835660 2577 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835664 2577 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835668 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835672 2577 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835677 2577 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835681 2577 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835685 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835688 2577 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835692 2577 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835696 2577 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835708 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835713 2577 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835717 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835728 2577 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835733 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835737 2577 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:00:27.847741 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835741 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835747 2577 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835752 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835756 2577 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835761 2577 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835765 2577 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835769 2577 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835773 2577 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835777 2577 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835781 2577 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835785 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835789 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835793 2577 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835797 2577 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835801 2577 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835806 2577 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835810 2577 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835814 2577 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835818 2577 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835822 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:00:27.848207 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835826 2577 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835830 2577 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835835 2577 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835841 2577 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835847 2577 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835852 2577 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835856 2577 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835868 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835874 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835878 2577 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835884 2577 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835889 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835893 2577 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835899 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835903 2577 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835907 2577 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835911 2577 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835915 2577 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835920 2577 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:00:27.848764 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835924 2577 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835928 2577 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835932 2577 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835936 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835940 2577 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835945 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835950 2577 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835954 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835959 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835963 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835967 2577 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835971 2577 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835976 2577 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835980 2577 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835984 2577 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835988 2577 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835992 2577 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.835996 2577 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836000 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:00:27.849210 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836004 2577 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836008 2577 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836023 2577 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836027 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836031 2577 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836035 2577 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836039 2577 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836045 2577 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836050 2577 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836054 2577 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836058 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836062 2577 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836066 2577 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836070 2577 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836074 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836078 2577 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836082 2577 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836086 2577 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836089 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836096 2577 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:00:27.849774 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836100 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:00:27.850282 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836112 2577 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:00:27.850282 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836117 2577 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:00:27.850282 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836121 2577 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:00:27.850282 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836126 2577 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:00:27.850282 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836132 2577 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:00:27.850282 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836139 2577 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:00:27.850282 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836143 2577 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:00:27.850282 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:27.836147 2577 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:00:27.850282 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.836236 2577 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 18:00:27.850282 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.839980 2577 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 18:00:27.850282 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.841049 2577 server.go:1014] "Starting client certificate rotation" Oct 30 18:00:27.850282 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.841159 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:00:27.850282 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.841864 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:00:27.850282 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.844040 2577 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:00:27.850282 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.848539 2577 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:00:27.859613 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.859594 2577 log.go:25] "Validated CRI v1 runtime API" Oct 30 18:00:27.863043 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.863028 2577 log.go:25] "Validated CRI v1 image API" Oct 30 18:00:27.863862 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.863834 2577 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 18:00:27.866273 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.866251 2577 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b4cf3e42-5869-4504-aa36-c7470c8ef023:/dev/nvme0n1p4 bb9e4620-dff8-49c9-b0d0-97064b75932a:/dev/nvme0n1p3] Oct 30 18:00:27.866321 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.866274 2577 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 18:00:27.871186 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.871075 2577 manager.go:217] Machine: {Timestamp:2026-10-30 18:00:27.869826715 +0000 UTC m=+0.324775613 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3098910 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23c59feae8cfc0ab014e430d2a0be8 SystemUUID:ec23c59f-eae8-cfc0-ab01-4e430d2a0be8 BootID:7b13398f-400a-49ec-9a05-0ad3464b240b Filesystems:[{Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ed:7c:0e:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ed:7c:0e:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ae:4a:32:e4:f2:53 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 18:00:27.871186 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.871181 2577 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 18:00:27.871310 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.871251 2577 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 18:00:27.872905 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.872880 2577 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 18:00:27.873038 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.872907 2577 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-106.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 18:00:27.873088 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.873040 2577 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 18:00:27.873088 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.873048 2577 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 18:00:27.873088 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.873079 2577 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:00:27.873168 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.873100 2577 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:00:27.873847 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.873837 2577 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 18:00:27.876322 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.876310 2577 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 18:00:27.876446 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.876437 2577 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 18:00:27.877181 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.877172 2577 kubelet.go:493] "Attempting to sync node with API server" Oct 30 18:00:27.877211 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.877185 2577 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 18:00:27.877211 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.877195 2577 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 18:00:27.877211 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.877204 2577 kubelet.go:405] "Adding apiserver pod source" Oct 30 18:00:27.877298 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.877213 2577 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 18:00:27.877652 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.877638 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:00:27.879976 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.879958 2577 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 18:00:27.884146 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.884130 2577 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:00:27.884977 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.884959 2577 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 18:00:27.885017 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.884996 2577 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 18:00:27.885194 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.885182 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 18:00:27.885222 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.885204 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 18:00:27.885222 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.885215 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 18:00:27.885273 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.885224 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 18:00:27.885273 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.885233 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 18:00:27.885273 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.885241 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 18:00:27.885668 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.885656 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 18:00:27.885698 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.885678 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 18:00:27.885698 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.885686 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 18:00:27.885698 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.885692 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 18:00:27.885774 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.885704 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 18:00:27.885774 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.885714 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 18:00:27.886386 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.886378 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 18:00:27.886386 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.886387 2577 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 18:00:27.892303 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.892284 2577 server.go:1263] "Started kubelet" Oct 30 18:00:27.892383 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.892355 2577 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 18:00:27.892529 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.892466 2577 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 18:00:27.892637 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.892540 2577 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 18:00:27.893207 ip-10-0-141-106 systemd[1]: Started Kubernetes Kubelet. Oct 30 18:00:27.893556 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.893535 2577 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 18:00:27.894323 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.894301 2577 server.go:317] "Adding debug handlers to kubelet server" Oct 30 18:00:27.897243 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:27.897222 2577 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 18:00:27.898885 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.898873 2577 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 18:00:27.898934 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.898891 2577 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 18:00:27.899015 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:27.898994 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Oct 30 18:00:27.899401 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.899386 2577 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 18:00:27.899401 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.899390 2577 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 18:00:27.899401 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.899433 2577 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 18:00:27.899544 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.899482 2577 reconstruct.go:97] "Volume reconstruction finished" Oct 30 18:00:27.899544 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.899492 2577 reconciler.go:29] "Reconciler: start to sync state" Oct 30 18:00:27.900008 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.899964 2577 factory.go:55] Registering systemd factory Oct 30 18:00:27.900008 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.899982 2577 factory.go:223] Registration of the systemd container factory successfully Oct 30 18:00:27.900185 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.900170 2577 factory.go:153] Registering CRI-O factory Oct 30 18:00:27.900185 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.900186 2577 factory.go:223] Registration of the crio container factory successfully Oct 30 18:00:27.900278 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.900225 2577 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 18:00:27.900278 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.900240 2577 factory.go:103] Registering Raw factory Oct 30 18:00:27.900278 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.900252 2577 manager.go:1199] Started watching for new ooms in manager Oct 30 18:00:27.901090 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.901067 2577 manager.go:319] Starting recovery of all containers Oct 30 18:00:27.911849 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.911831 2577 manager.go:324] Recovery completed Oct 30 18:00:27.915079 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:27.915051 2577 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory Oct 30 18:00:27.918568 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.918488 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:27.920152 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.920133 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:27.920226 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.920169 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:27.920226 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.920183 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:27.920709 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.920692 2577 cpu_manager.go:226] "Starting" policy="none" Oct 30 18:00:27.920709 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.920707 2577 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 18:00:27.920814 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.920694 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:27.920814 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.920730 2577 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 18:00:27.921392 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.921377 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:27.921463 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.921409 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:27.921463 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.921435 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:27.921846 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.921831 2577 policy_none.go:50] "Start" Oct 30 18:00:27.921846 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.921849 2577 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 18:00:27.921970 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.921862 2577 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 18:00:27.922938 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.922924 2577 policy_none.go:44] "Start" Oct 30 18:00:27.926194 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.926173 2577 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-106.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 18:00:27.926275 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:27.926188 2577 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-106.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 18:00:27.927177 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:27.926196 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-106.ec2.internal.18e3608b44a92630 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-106.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-106.ec2.internal,},FirstTimestamp:2026-10-30 18:00:27.892246064 +0000 UTC m=+0.347194962,LastTimestamp:2026-10-30 18:00:27.892246064 +0000 UTC m=+0.347194962,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-106.ec2.internal,}" Oct 30 18:00:27.941087 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:27.940849 2577 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-106.ec2.internal.18e3608b4652f30e default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-106.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-106.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-106.ec2.internal,},FirstTimestamp:2026-10-30 18:00:27.92015131 +0000 UTC m=+0.375100214,LastTimestamp:2026-10-30 18:00:27.92015131 +0000 UTC m=+0.375100214,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-106.ec2.internal,}" Oct 30 18:00:27.941220 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:27.941011 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 18:00:27.941220 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:27.941177 2577 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-106.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 18:00:27.943874 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.943851 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 18:00:27.944004 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.943891 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sm95g" Oct 30 18:00:27.948707 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.948688 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-sm95g" Oct 30 18:00:27.966013 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.965986 2577 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 18:00:27.968287 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.968272 2577 manager.go:341] "Starting Device Plugin manager" Oct 30 18:00:27.968373 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:27.968301 2577 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 18:00:27.968373 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.968309 2577 server.go:87] "Starting device plugin registration server" Oct 30 18:00:27.968534 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.968521 2577 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 18:00:27.968589 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.968537 2577 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 18:00:27.968724 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.968697 2577 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 18:00:27.968822 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.968780 2577 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 18:00:27.968822 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.968790 2577 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 18:00:27.969274 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:27.969254 2577 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 18:00:27.969351 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:27.969306 2577 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-106.ec2.internal\" not found" Oct 30 18:00:27.998523 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:27.988807 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:28.022646 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.022622 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 18:00:28.023977 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.023962 2577 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 18:00:28.024039 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.023982 2577 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 18:00:28.024039 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.023998 2577 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 18:00:28.024039 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:28.024029 2577 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 18:00:28.036105 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.036091 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:28.069347 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.069302 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:28.070230 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.070215 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:28.070317 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.070247 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:28.070317 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.070286 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:28.070445 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.070325 2577 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-106.ec2.internal" Oct 30 18:00:28.086836 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.086818 2577 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-106.ec2.internal" Oct 30 18:00:28.086901 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:28.086839 2577 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-106.ec2.internal\": node \"ip-10-0-141-106.ec2.internal\" not found" Oct 30 18:00:28.121943 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:28.121926 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Oct 30 18:00:28.125010 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.124990 2577 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal"] Oct 30 18:00:28.125069 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.125042 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:28.125779 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.125764 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:28.125870 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.125802 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:28.125870 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.125817 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:28.127837 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.127822 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:28.127986 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.127972 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:00:28.128034 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.128000 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:28.128511 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.128496 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:28.128585 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.128520 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:28.128585 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.128535 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:28.128585 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.128574 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:28.128723 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.128600 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:28.128723 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.128609 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:28.130461 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.130445 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Oct 30 18:00:28.130531 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.130468 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:28.131123 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.131109 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:28.131172 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.131132 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:28.131172 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.131143 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:28.161678 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:28.161660 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-106.ec2.internal\" not found" node="ip-10-0-141-106.ec2.internal" Oct 30 18:00:28.163432 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:28.163396 2577 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-106.ec2.internal\" not found" node="ip-10-0-141-106.ec2.internal" Oct 30 18:00:28.200594 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.200571 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ace45daccc39fb3be3eb84e89404869-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" (UID: \"2ace45daccc39fb3be3eb84e89404869\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:00:28.200683 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.200607 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ace45daccc39fb3be3eb84e89404869-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" (UID: \"2ace45daccc39fb3be3eb84e89404869\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:00:28.200683 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.200634 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1bdd65ea12e9d18ae4c0eda6db4fb016-config\") pod \"kube-apiserver-proxy-ip-10-0-141-106.ec2.internal\" (UID: \"1bdd65ea12e9d18ae4c0eda6db4fb016\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Oct 30 18:00:28.222038 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:28.222019 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Oct 30 18:00:28.301230 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.301207 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1bdd65ea12e9d18ae4c0eda6db4fb016-config\") pod \"kube-apiserver-proxy-ip-10-0-141-106.ec2.internal\" (UID: \"1bdd65ea12e9d18ae4c0eda6db4fb016\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Oct 30 18:00:28.301315 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.301232 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ace45daccc39fb3be3eb84e89404869-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" (UID: \"2ace45daccc39fb3be3eb84e89404869\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:00:28.301315 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.301250 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ace45daccc39fb3be3eb84e89404869-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" (UID: \"2ace45daccc39fb3be3eb84e89404869\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:00:28.301315 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.301300 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ace45daccc39fb3be3eb84e89404869-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" (UID: \"2ace45daccc39fb3be3eb84e89404869\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:00:28.301315 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.301300 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1bdd65ea12e9d18ae4c0eda6db4fb016-config\") pod \"kube-apiserver-proxy-ip-10-0-141-106.ec2.internal\" (UID: \"1bdd65ea12e9d18ae4c0eda6db4fb016\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Oct 30 18:00:28.301315 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.301306 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ace45daccc39fb3be3eb84e89404869-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" (UID: \"2ace45daccc39fb3be3eb84e89404869\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:00:28.322549 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:28.322514 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Oct 30 18:00:28.422595 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:28.422563 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Oct 30 18:00:28.465747 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.465728 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:00:28.466817 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.466802 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Oct 30 18:00:28.522839 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:28.522816 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Oct 30 18:00:28.623323 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:28.623264 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Oct 30 18:00:28.723740 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:28.723719 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Oct 30 18:00:28.824322 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:28.824295 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Oct 30 18:00:28.840488 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.840458 2577 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 18:00:28.840605 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.840589 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 18:00:28.840645 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.840620 2577 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 18:00:28.925310 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:28.925274 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Oct 30 18:00:28.952169 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.952135 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:55:29 +0000 UTC" deadline="2028-05-15 20:54:07.59113548 +0000 UTC" Oct 30 18:00:28.952169 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.952164 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13514h53m38.638974311s" Oct 30 18:00:28.956295 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.956279 2577 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:00:28.958718 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.958703 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:00:28.958776 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.958769 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:00:28.958809 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:28.958782 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:00:29.011301 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:29.011275 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1bdd65ea12e9d18ae4c0eda6db4fb016.slice/crio-0c1b111e1b4322013cec7a18f50d9149c78abb670cb2e693767430c964ffc492 WatchSource:0}: Error finding container 0c1b111e1b4322013cec7a18f50d9149c78abb670cb2e693767430c964ffc492: Status 404 returned error can't find the container with id 0c1b111e1b4322013cec7a18f50d9149c78abb670cb2e693767430c964ffc492 Oct 30 18:00:29.021541 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:29.021523 2577 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:00:29.025551 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:29.025531 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Oct 30 18:00:29.027572 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:29.027550 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ace45daccc39fb3be3eb84e89404869.slice/crio-ed434e77d9eaa7fcb359f40da57c50494b5b63005a9e002add9d7ee20df0a13c WatchSource:0}: Error finding container ed434e77d9eaa7fcb359f40da57c50494b5b63005a9e002add9d7ee20df0a13c: Status 404 returned error can't find the container with id ed434e77d9eaa7fcb359f40da57c50494b5b63005a9e002add9d7ee20df0a13c Oct 30 18:00:29.029329 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:29.029250 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" event={"ID":"1bdd65ea12e9d18ae4c0eda6db4fb016","Type":"ContainerStarted","Data":"0c1b111e1b4322013cec7a18f50d9149c78abb670cb2e693767430c964ffc492"} Oct 30 18:00:29.126213 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:29.126180 2577 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Oct 30 18:00:29.154670 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:29.154648 2577 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:29.199715 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:29.199663 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Oct 30 18:00:29.223007 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:29.222986 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:00:29.223099 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:29.223079 2577 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:00:29.245886 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:29.245866 2577 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:00:29.520877 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:29.520803 2577 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:29.878398 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:29.878327 2577 apiserver.go:52] "Watching apiserver" Oct 30 18:00:29.900436 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:29.900394 2577 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 18:00:31.121435 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.121406 2577 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 18:00:31.121852 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.121826 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/multus-additional-cni-plugins-mw76k","openshift-network-diagnostics/network-check-target-94r4x","openshift-image-registry/node-ca-h6lp9","openshift-multus/network-metrics-daemon-g8h9t","openshift-ovn-kubernetes/ovnkube-node-m556r","kube-system/global-pull-secret-syncer-brgpf","openshift-network-operator/iptables-alerter-t7sqv","kube-system/konnectivity-agent-zjlpz","kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5","openshift-multus/multus-9bq58","openshift-cluster-node-tuning-operator/tuned-k8684","openshift-dns/node-resolver-td6vr","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal"] Oct 30 18:00:31.124548 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.124526 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t7sqv" Oct 30 18:00:31.126795 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.126774 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-td6vr" Oct 30 18:00:31.129187 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.129170 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h6lp9" Oct 30 18:00:31.131597 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.131577 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:31.131680 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.131650 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:00:31.133608 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.133591 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.136067 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.136045 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:31.136169 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.136118 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:00:31.138846 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.138329 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zjlpz" Oct 30 18:00:31.141420 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.140875 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.141420 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.141405 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:31.141600 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.141455 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ljv2v\"" Oct 30 18:00:31.141600 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.141470 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 18:00:31.141600 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.141538 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 18:00:31.145974 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.145952 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.146072 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.145980 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l5dm7\"" Oct 30 18:00:31.146139 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.146095 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 18:00:31.146194 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.146186 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 18:00:31.146366 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.146351 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:31.146464 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.146436 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:00:31.149463 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.149441 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.150240 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.150220 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 18:00:31.150240 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.150232 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 18:00:31.150465 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.150448 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dzg75\"" Oct 30 18:00:31.150582 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.150560 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 18:00:31.151837 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.151818 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.153609 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.153592 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 18:00:31.154059 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.154036 2577 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 18:00:31.160548 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.160501 2577 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-8jbn5" Oct 30 18:00:31.168528 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.167868 2577 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-8jbn5" Oct 30 18:00:31.197107 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.197091 2577 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 18:00:31.205083 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205060 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75xlp\" (UniqueName: \"kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp\") pod \"network-check-target-94r4x\" (UID: \"a3c3fe6d-7fab-4cea-a04e-693b985ceb49\") " pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:31.205196 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205090 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.205196 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205111 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-slash\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.205196 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205133 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40dd1611-7ec9-411f-920d-e79df7273b90-konnectivity-ca\") pod \"konnectivity-agent-zjlpz\" (UID: \"40dd1611-7ec9-411f-920d-e79df7273b90\") " pod="kube-system/konnectivity-agent-zjlpz" Oct 30 18:00:31.205310 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205193 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-registration-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.205310 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205224 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-multus-cni-dir\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.205310 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205244 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-run-k8s-cni-cncf-io\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.205310 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205259 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-node-log\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.205310 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205301 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-cni-bin\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.205536 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205332 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d458e1b-228c-4e8e-b66f-9688583b39db-ovnkube-config\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.205536 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205364 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d458e1b-228c-4e8e-b66f-9688583b39db-ovn-node-metrics-cert\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.205536 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205391 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-socket-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.205536 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205414 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-var-lib-cni-bin\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.205536 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205446 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-var-lib-kubelet\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.205536 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205489 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs\") pod \"network-metrics-daemon-g8h9t\" (UID: \"914c8c09-aa3d-4543-9d91-fa443a09b1ad\") " pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:31.205791 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205543 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-modprobe-d\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.205791 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205567 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsj7n\" (UniqueName: \"kubernetes.io/projected/5e5fca56-5e6b-404d-84b2-4a876f5034cd-kube-api-access-rsj7n\") pod \"iptables-alerter-t7sqv\" (UID: \"5e5fca56-5e6b-404d-84b2-4a876f5034cd\") " pod="openshift-network-operator/iptables-alerter-t7sqv" Oct 30 18:00:31.205791 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205583 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d711c2d-0597-4d83-a884-83e50d98378c-hosts-file\") pod \"node-resolver-td6vr\" (UID: \"8d711c2d-0597-4d83-a884-83e50d98378c\") " pod="openshift-dns/node-resolver-td6vr" Oct 30 18:00:31.205791 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205636 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d711c2d-0597-4d83-a884-83e50d98378c-tmp-dir\") pod \"node-resolver-td6vr\" (UID: \"8d711c2d-0597-4d83-a884-83e50d98378c\") " pod="openshift-dns/node-resolver-td6vr" Oct 30 18:00:31.205791 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205667 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-etc-openvswitch\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.205791 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205694 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pm2mh\" (UniqueName: \"kubernetes.io/projected/6d458e1b-228c-4e8e-b66f-9688583b39db-kube-api-access-pm2mh\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.205791 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205728 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-lib-modules\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.205791 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205769 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-var-lib-kubelet\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.206137 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205803 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-dbus\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:31.206137 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205840 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-system-cni-dir\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.206137 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205867 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-run-netns\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.206137 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205893 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d458e1b-228c-4e8e-b66f-9688583b39db-env-overrides\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.206137 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205927 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d458e1b-228c-4e8e-b66f-9688583b39db-ovnkube-script-lib\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.206137 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205974 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:31.206137 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.205993 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-system-cni-dir\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.206137 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206030 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pkx2s\" (UniqueName: \"kubernetes.io/projected/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-kube-api-access-pkx2s\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.206137 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206078 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-var-lib-openvswitch\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.206137 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206122 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-run-ovn\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.206137 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206142 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-run-ovn-kubernetes\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.206536 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206169 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-sysctl-d\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.206536 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206206 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11311f90-d76a-42b5-b6a2-35c0b62bd222-serviceca\") pod \"node-ca-h6lp9\" (UID: \"11311f90-d76a-42b5-b6a2-35c0b62bd222\") " pod="openshift-image-registry/node-ca-h6lp9" Oct 30 18:00:31.206536 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206248 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfmlm\" (UniqueName: \"kubernetes.io/projected/11311f90-d76a-42b5-b6a2-35c0b62bd222-kube-api-access-bfmlm\") pod \"node-ca-h6lp9\" (UID: \"11311f90-d76a-42b5-b6a2-35c0b62bd222\") " pod="openshift-image-registry/node-ca-h6lp9" Oct 30 18:00:31.206536 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206268 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-sys-fs\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.206536 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206290 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-multus-socket-dir-parent\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.206536 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206327 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.206536 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206356 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40dd1611-7ec9-411f-920d-e79df7273b90-agent-certs\") pod \"konnectivity-agent-zjlpz\" (UID: \"40dd1611-7ec9-411f-920d-e79df7273b90\") " pod="kube-system/konnectivity-agent-zjlpz" Oct 30 18:00:31.206536 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206390 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-run\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.206536 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-host\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.206536 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206441 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p496j\" (UniqueName: \"kubernetes.io/projected/76146060-cfc6-4765-82a3-5c5c7601dfda-kube-api-access-p496j\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.206536 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206492 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.206908 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206544 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8754b620-b515-4ee4-9055-469a81126c9e-cni-binary-copy\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.206908 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206572 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-hostroot\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.206908 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206594 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-etc-kubernetes\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.206908 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206617 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-cni-netd\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.206908 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206645 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-sys\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.206908 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206681 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-device-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.206908 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206707 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e5fca56-5e6b-404d-84b2-4a876f5034cd-host-slash\") pod \"iptables-alerter-t7sqv\" (UID: \"5e5fca56-5e6b-404d-84b2-4a876f5034cd\") " pod="openshift-network-operator/iptables-alerter-t7sqv" Oct 30 18:00:31.206908 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206731 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-log-socket\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.206908 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206771 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5e5fca56-5e6b-404d-84b2-4a876f5034cd-iptables-alerter-script\") pod \"iptables-alerter-t7sqv\" (UID: \"5e5fca56-5e6b-404d-84b2-4a876f5034cd\") " pod="openshift-network-operator/iptables-alerter-t7sqv" Oct 30 18:00:31.206908 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206800 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-sysctl-conf\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.206908 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206839 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-cnibin\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.206908 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206869 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8754b620-b515-4ee4-9055-469a81126c9e-multus-daemon-config\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.206908 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206892 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-run-multus-certs\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.207338 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.206942 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dtfcr\" (UniqueName: \"kubernetes.io/projected/914c8c09-aa3d-4543-9d91-fa443a09b1ad-kube-api-access-dtfcr\") pod \"network-metrics-daemon-g8h9t\" (UID: \"914c8c09-aa3d-4543-9d91-fa443a09b1ad\") " pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:31.207338 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207058 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-run-netns\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.207338 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207102 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-sysconfig\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.207338 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207126 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-tuned\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.207338 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207147 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11311f90-d76a-42b5-b6a2-35c0b62bd222-host\") pod \"node-ca-h6lp9\" (UID: \"11311f90-d76a-42b5-b6a2-35c0b62bd222\") " pod="openshift-image-registry/node-ca-h6lp9" Oct 30 18:00:31.207338 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207169 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-systemd-units\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.207338 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207256 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-run-systemd\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.207338 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207291 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-var-lib-cni-multus\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.207338 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207323 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nstb\" (UniqueName: \"kubernetes.io/projected/8754b620-b515-4ee4-9055-469a81126c9e-kube-api-access-2nstb\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.207710 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207345 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-kubernetes\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.207710 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207369 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-kubelet-config\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:31.207710 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207420 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.207710 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207456 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbr88\" (UniqueName: \"kubernetes.io/projected/8d711c2d-0597-4d83-a884-83e50d98378c-kube-api-access-rbr88\") pod \"node-resolver-td6vr\" (UID: \"8d711c2d-0597-4d83-a884-83e50d98378c\") " pod="openshift-dns/node-resolver-td6vr" Oct 30 18:00:31.207710 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207530 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-kubelet\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.207710 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207565 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-run-openvswitch\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.207710 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207587 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.207710 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207608 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-multus-conf-dir\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.207710 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207645 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76146060-cfc6-4765-82a3-5c5c7601dfda-tmp\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.207710 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207673 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.207710 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207706 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-etc-selinux\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.208185 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207754 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4xwn\" (UniqueName: \"kubernetes.io/projected/4627766d-20a1-46cd-baf9-cff7cc120679-kube-api-access-q4xwn\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.208185 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207788 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-cnibin\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.208185 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.207821 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-systemd\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.227449 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.227422 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" event={"ID":"2ace45daccc39fb3be3eb84e89404869","Type":"ContainerStarted","Data":"ed434e77d9eaa7fcb359f40da57c50494b5b63005a9e002add9d7ee20df0a13c"} Oct 30 18:00:31.241720 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.241699 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 18:00:31.241829 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.241779 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 18:00:31.241961 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.241938 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 18:00:31.242023 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.241984 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 18:00:31.242196 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.242176 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 18:00:31.242320 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.242218 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 18:00:31.242878 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.242864 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 18:00:31.242973 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.242956 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 18:00:31.243058 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.243037 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tlc5r\"" Oct 30 18:00:31.243160 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.243127 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:00:31.243218 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.243162 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 18:00:31.243270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.243237 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hdw4l\"" Oct 30 18:00:31.243366 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.243352 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 18:00:31.243451 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.243431 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 18:00:31.243578 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.243536 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 18:00:31.243719 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.243703 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mj45s\"" Oct 30 18:00:31.243806 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.243747 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 18:00:31.243806 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.243778 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 18:00:31.243906 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.243820 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 18:00:31.243906 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.243848 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hj78h\"" Oct 30 18:00:31.243906 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.243867 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 18:00:31.244043 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.243965 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 18:00:31.244043 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.243999 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b72w5\"" Oct 30 18:00:31.244111 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.244052 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8z8qv\"" Oct 30 18:00:31.308710 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.308679 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-sysctl-d\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.308862 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.308719 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11311f90-d76a-42b5-b6a2-35c0b62bd222-serviceca\") pod \"node-ca-h6lp9\" (UID: \"11311f90-d76a-42b5-b6a2-35c0b62bd222\") " pod="openshift-image-registry/node-ca-h6lp9" Oct 30 18:00:31.308862 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.308747 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfmlm\" (UniqueName: \"kubernetes.io/projected/11311f90-d76a-42b5-b6a2-35c0b62bd222-kube-api-access-bfmlm\") pod \"node-ca-h6lp9\" (UID: \"11311f90-d76a-42b5-b6a2-35c0b62bd222\") " pod="openshift-image-registry/node-ca-h6lp9" Oct 30 18:00:31.308862 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.308773 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-sys-fs\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.308862 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.308806 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-multus-socket-dir-parent\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.308862 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.308829 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.308862 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.308855 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40dd1611-7ec9-411f-920d-e79df7273b90-agent-certs\") pod \"konnectivity-agent-zjlpz\" (UID: \"40dd1611-7ec9-411f-920d-e79df7273b90\") " pod="kube-system/konnectivity-agent-zjlpz" Oct 30 18:00:31.309160 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.308876 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-run\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.309160 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.308896 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-host\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.309160 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.308919 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p496j\" (UniqueName: \"kubernetes.io/projected/76146060-cfc6-4765-82a3-5c5c7601dfda-kube-api-access-p496j\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.309160 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.308942 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.309160 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.308964 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8754b620-b515-4ee4-9055-469a81126c9e-cni-binary-copy\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.309160 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.308994 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-hostroot\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.309160 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309017 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-etc-kubernetes\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.309160 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309038 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-cni-netd\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.309160 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309058 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-sys\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.309160 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309080 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-device-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.309160 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309104 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e5fca56-5e6b-404d-84b2-4a876f5034cd-host-slash\") pod \"iptables-alerter-t7sqv\" (UID: \"5e5fca56-5e6b-404d-84b2-4a876f5034cd\") " pod="openshift-network-operator/iptables-alerter-t7sqv" Oct 30 18:00:31.309160 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309127 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-log-socket\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.309160 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309149 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5e5fca56-5e6b-404d-84b2-4a876f5034cd-iptables-alerter-script\") pod \"iptables-alerter-t7sqv\" (UID: \"5e5fca56-5e6b-404d-84b2-4a876f5034cd\") " pod="openshift-network-operator/iptables-alerter-t7sqv" Oct 30 18:00:31.309802 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309174 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-sysctl-conf\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.309802 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309197 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-cnibin\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.309802 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309216 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8754b620-b515-4ee4-9055-469a81126c9e-multus-daemon-config\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.309802 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-run-multus-certs\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.309802 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309246 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtfcr\" (UniqueName: \"kubernetes.io/projected/914c8c09-aa3d-4543-9d91-fa443a09b1ad-kube-api-access-dtfcr\") pod \"network-metrics-daemon-g8h9t\" (UID: \"914c8c09-aa3d-4543-9d91-fa443a09b1ad\") " pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:31.309802 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309259 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-run-netns\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.309802 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309271 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-sysconfig\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.309802 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309284 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-tuned\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.309802 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309298 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11311f90-d76a-42b5-b6a2-35c0b62bd222-host\") pod \"node-ca-h6lp9\" (UID: \"11311f90-d76a-42b5-b6a2-35c0b62bd222\") " pod="openshift-image-registry/node-ca-h6lp9" Oct 30 18:00:31.309802 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309315 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-systemd-units\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.309802 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309334 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-run-systemd\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.309802 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309355 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-var-lib-cni-multus\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.309802 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309373 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nstb\" (UniqueName: \"kubernetes.io/projected/8754b620-b515-4ee4-9055-469a81126c9e-kube-api-access-2nstb\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.309802 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309397 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-kubernetes\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.309802 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309433 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-kubelet-config\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:31.309802 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.309802 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309482 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbr88\" (UniqueName: \"kubernetes.io/projected/8d711c2d-0597-4d83-a884-83e50d98378c-kube-api-access-rbr88\") pod \"node-resolver-td6vr\" (UID: \"8d711c2d-0597-4d83-a884-83e50d98378c\") " pod="openshift-dns/node-resolver-td6vr" Oct 30 18:00:31.310572 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309496 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-kubelet\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.310572 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309534 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-run-openvswitch\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.310572 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309560 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.310572 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309584 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-multus-conf-dir\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.310572 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309605 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76146060-cfc6-4765-82a3-5c5c7601dfda-tmp\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.310572 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309623 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.310572 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309641 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-etc-selinux\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.310572 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309659 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4xwn\" (UniqueName: \"kubernetes.io/projected/4627766d-20a1-46cd-baf9-cff7cc120679-kube-api-access-q4xwn\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.310572 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-cnibin\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.310572 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309696 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-systemd\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.310572 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309727 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xlp\" (UniqueName: \"kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp\") pod \"network-check-target-94r4x\" (UID: \"a3c3fe6d-7fab-4cea-a04e-693b985ceb49\") " pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:31.310572 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309755 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.310572 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309790 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-slash\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.310572 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309814 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40dd1611-7ec9-411f-920d-e79df7273b90-konnectivity-ca\") pod \"konnectivity-agent-zjlpz\" (UID: \"40dd1611-7ec9-411f-920d-e79df7273b90\") " pod="kube-system/konnectivity-agent-zjlpz" Oct 30 18:00:31.310572 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309835 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-registration-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.310572 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309860 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-multus-cni-dir\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.310572 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309879 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-run-k8s-cni-cncf-io\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.311337 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309893 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-node-log\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.311337 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309909 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-cni-bin\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.311337 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309924 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d458e1b-228c-4e8e-b66f-9688583b39db-ovnkube-config\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.311337 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309940 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d458e1b-228c-4e8e-b66f-9688583b39db-ovn-node-metrics-cert\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.311337 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309955 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-socket-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.311337 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.309980 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-var-lib-cni-bin\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.311337 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310034 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-var-lib-kubelet\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.311337 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310061 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs\") pod \"network-metrics-daemon-g8h9t\" (UID: \"914c8c09-aa3d-4543-9d91-fa443a09b1ad\") " pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:31.311337 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310087 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-modprobe-d\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.311337 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310112 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsj7n\" (UniqueName: \"kubernetes.io/projected/5e5fca56-5e6b-404d-84b2-4a876f5034cd-kube-api-access-rsj7n\") pod \"iptables-alerter-t7sqv\" (UID: \"5e5fca56-5e6b-404d-84b2-4a876f5034cd\") " pod="openshift-network-operator/iptables-alerter-t7sqv" Oct 30 18:00:31.311337 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310137 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d711c2d-0597-4d83-a884-83e50d98378c-hosts-file\") pod \"node-resolver-td6vr\" (UID: \"8d711c2d-0597-4d83-a884-83e50d98378c\") " pod="openshift-dns/node-resolver-td6vr" Oct 30 18:00:31.311337 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310153 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d711c2d-0597-4d83-a884-83e50d98378c-tmp-dir\") pod \"node-resolver-td6vr\" (UID: \"8d711c2d-0597-4d83-a884-83e50d98378c\") " pod="openshift-dns/node-resolver-td6vr" Oct 30 18:00:31.311337 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310178 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-etc-openvswitch\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.311337 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310203 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm2mh\" (UniqueName: \"kubernetes.io/projected/6d458e1b-228c-4e8e-b66f-9688583b39db-kube-api-access-pm2mh\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.311337 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310230 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-lib-modules\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.311337 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310258 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-var-lib-kubelet\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.311337 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310282 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-dbus\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:31.312117 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310310 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-system-cni-dir\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.312117 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310336 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-run-netns\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.312117 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310360 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d458e1b-228c-4e8e-b66f-9688583b39db-env-overrides\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.312117 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310387 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d458e1b-228c-4e8e-b66f-9688583b39db-ovnkube-script-lib\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.312117 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310414 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:31.312117 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310440 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-system-cni-dir\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.312117 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310466 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkx2s\" (UniqueName: \"kubernetes.io/projected/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-kube-api-access-pkx2s\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.312117 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310488 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-var-lib-openvswitch\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.312117 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310520 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-run-ovn\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.312117 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310537 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-run-ovn-kubernetes\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.312117 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310639 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-run-ovn-kubernetes\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.312117 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.310774 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-sysctl-d\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.312117 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.311248 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11311f90-d76a-42b5-b6a2-35c0b62bd222-serviceca\") pod \"node-ca-h6lp9\" (UID: \"11311f90-d76a-42b5-b6a2-35c0b62bd222\") " pod="openshift-image-registry/node-ca-h6lp9" Oct 30 18:00:31.312117 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.311549 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-sys-fs\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.312117 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.311668 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-multus-socket-dir-parent\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.312117 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.311712 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.312117 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.312004 2577 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 18:00:31.312766 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.312346 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-etc-selinux\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.312766 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.312363 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-run\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.312766 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.312434 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-kubelet-config\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:31.312766 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.312491 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-host\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.312766 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.312594 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-run-netns\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.312766 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.312645 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-sysconfig\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.312952 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.312917 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.315270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.312983 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-multus-conf-dir\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.315270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.313039 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-run-k8s-cni-cncf-io\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.315270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.313098 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-node-log\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.315270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.313133 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.315270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.313223 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-cnibin\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.315270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.313282 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-systemd\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.315270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.313657 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d458e1b-228c-4e8e-b66f-9688583b39db-ovnkube-config\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.315270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.314008 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.315270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.314119 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8754b620-b515-4ee4-9055-469a81126c9e-cni-binary-copy\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.315270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.314208 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-modprobe-d\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.315270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.314255 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-hostroot\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.315270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.314271 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-kubelet\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.315270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.314298 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-etc-kubernetes\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.315270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.314320 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-run-openvswitch\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.315270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.314343 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-cni-netd\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.315270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.314389 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d711c2d-0597-4d83-a884-83e50d98378c-hosts-file\") pod \"node-resolver-td6vr\" (UID: \"8d711c2d-0597-4d83-a884-83e50d98378c\") " pod="openshift-dns/node-resolver-td6vr" Oct 30 18:00:31.315270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.314436 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e5fca56-5e6b-404d-84b2-4a876f5034cd-host-slash\") pod \"iptables-alerter-t7sqv\" (UID: \"5e5fca56-5e6b-404d-84b2-4a876f5034cd\") " pod="openshift-network-operator/iptables-alerter-t7sqv" Oct 30 18:00:31.315270 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.314456 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.316081 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.314480 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-device-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.316081 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.314534 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-log-socket\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.316081 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.314571 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:31.316081 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.314634 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret podName:b0df4b9d-89ce-4bb4-b759-68f1b499b52a nodeName:}" failed. No retries permitted until 2026-10-30 18:00:31.814616068 +0000 UTC m=+3.073474831 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret") pod "global-pull-secret-syncer-brgpf" (UID: "b0df4b9d-89ce-4bb4-b759-68f1b499b52a") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:31.316081 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.314699 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d711c2d-0597-4d83-a884-83e50d98378c-tmp-dir\") pod \"node-resolver-td6vr\" (UID: \"8d711c2d-0597-4d83-a884-83e50d98378c\") " pod="openshift-dns/node-resolver-td6vr" Oct 30 18:00:31.316081 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.314766 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-etc-openvswitch\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.316081 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.314965 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-lib-modules\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.316081 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.315019 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-var-lib-kubelet\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.316081 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.314394 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-sys\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.316081 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.315078 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-system-cni-dir\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.316081 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.315115 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-run-netns\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.316699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316129 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5e5fca56-5e6b-404d-84b2-4a876f5034cd-iptables-alerter-script\") pod \"iptables-alerter-t7sqv\" (UID: \"5e5fca56-5e6b-404d-84b2-4a876f5034cd\") " pod="openshift-network-operator/iptables-alerter-t7sqv" Oct 30 18:00:31.316699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316132 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d458e1b-228c-4e8e-b66f-9688583b39db-env-overrides\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.316699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316199 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-var-lib-cni-bin\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.316699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316223 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-sysctl-conf\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.316699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316280 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-cnibin\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.316699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.312363 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-kubernetes\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.316699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.314118 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-dbus\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:31.316699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.313140 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-cni-bin\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.316699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316296 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-socket-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.316699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316346 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-system-cni-dir\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.316699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316365 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-run-multus-certs\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.316699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316282 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76146060-cfc6-4765-82a3-5c5c7601dfda-tmp\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.316699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316399 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40dd1611-7ec9-411f-920d-e79df7273b90-agent-certs\") pod \"konnectivity-agent-zjlpz\" (UID: \"40dd1611-7ec9-411f-920d-e79df7273b90\") " pod="kube-system/konnectivity-agent-zjlpz" Oct 30 18:00:31.316699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316460 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-var-lib-openvswitch\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.316699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316521 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-run-ovn\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.316699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316570 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-run-systemd\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.316699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316591 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11311f90-d76a-42b5-b6a2-35c0b62bd222-host\") pod \"node-ca-h6lp9\" (UID: \"11311f90-d76a-42b5-b6a2-35c0b62bd222\") " pod="openshift-image-registry/node-ca-h6lp9" Oct 30 18:00:31.316699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316611 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-systemd-units\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.317582 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316652 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-var-lib-kubelet\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.317582 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.316660 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:31.317582 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.316703 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs podName:914c8c09-aa3d-4543-9d91-fa443a09b1ad nodeName:}" failed. No retries permitted until 2026-10-30 18:00:31.816688305 +0000 UTC m=+3.075547054 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs") pod "network-metrics-daemon-g8h9t" (UID: "914c8c09-aa3d-4543-9d91-fa443a09b1ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:31.317582 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316703 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-multus-cni-dir\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.317582 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316757 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.317582 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316759 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-registration-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.317582 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316807 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-slash\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.317582 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.316809 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-var-lib-cni-multus\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.317582 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.317183 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8754b620-b515-4ee4-9055-469a81126c9e-multus-daemon-config\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.317582 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.317360 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40dd1611-7ec9-411f-920d-e79df7273b90-konnectivity-ca\") pod \"konnectivity-agent-zjlpz\" (UID: \"40dd1611-7ec9-411f-920d-e79df7273b90\") " pod="kube-system/konnectivity-agent-zjlpz" Oct 30 18:00:31.318493 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.318469 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d458e1b-228c-4e8e-b66f-9688583b39db-ovnkube-script-lib\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.318684 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.318663 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-tuned\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.322707 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.321352 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d458e1b-228c-4e8e-b66f-9688583b39db-ovn-node-metrics-cert\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.338717 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.338643 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:00:31.338717 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.338670 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:00:31.338717 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.338684 2577 projected.go:196] Error preparing data for projected volume kube-api-access-75xlp for pod openshift-network-diagnostics/network-check-target-94r4x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:31.338928 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.338866 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp podName:a3c3fe6d-7fab-4cea-a04e-693b985ceb49 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:31.838756133 +0000 UTC m=+3.097614877 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-75xlp" (UniqueName: "kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp") pod "network-check-target-94r4x" (UID: "a3c3fe6d-7fab-4cea-a04e-693b985ceb49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:31.341725 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.341702 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4xwn\" (UniqueName: \"kubernetes.io/projected/4627766d-20a1-46cd-baf9-cff7cc120679-kube-api-access-q4xwn\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.342327 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.342310 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfmlm\" (UniqueName: \"kubernetes.io/projected/11311f90-d76a-42b5-b6a2-35c0b62bd222-kube-api-access-bfmlm\") pod \"node-ca-h6lp9\" (UID: \"11311f90-d76a-42b5-b6a2-35c0b62bd222\") " pod="openshift-image-registry/node-ca-h6lp9" Oct 30 18:00:31.343291 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.342793 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm2mh\" (UniqueName: \"kubernetes.io/projected/6d458e1b-228c-4e8e-b66f-9688583b39db-kube-api-access-pm2mh\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.343291 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.343228 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsj7n\" (UniqueName: \"kubernetes.io/projected/5e5fca56-5e6b-404d-84b2-4a876f5034cd-kube-api-access-rsj7n\") pod \"iptables-alerter-t7sqv\" (UID: \"5e5fca56-5e6b-404d-84b2-4a876f5034cd\") " pod="openshift-network-operator/iptables-alerter-t7sqv" Oct 30 18:00:31.343291 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.343234 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nstb\" (UniqueName: \"kubernetes.io/projected/8754b620-b515-4ee4-9055-469a81126c9e-kube-api-access-2nstb\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.343291 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.343229 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkx2s\" (UniqueName: \"kubernetes.io/projected/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-kube-api-access-pkx2s\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.343706 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.343686 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtfcr\" (UniqueName: \"kubernetes.io/projected/914c8c09-aa3d-4543-9d91-fa443a09b1ad-kube-api-access-dtfcr\") pod \"network-metrics-daemon-g8h9t\" (UID: \"914c8c09-aa3d-4543-9d91-fa443a09b1ad\") " pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:31.345289 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.345225 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p496j\" (UniqueName: \"kubernetes.io/projected/76146060-cfc6-4765-82a3-5c5c7601dfda-kube-api-access-p496j\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.346090 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.346072 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbr88\" (UniqueName: \"kubernetes.io/projected/8d711c2d-0597-4d83-a884-83e50d98378c-kube-api-access-rbr88\") pod \"node-resolver-td6vr\" (UID: \"8d711c2d-0597-4d83-a884-83e50d98378c\") " pod="openshift-dns/node-resolver-td6vr" Oct 30 18:00:31.439560 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.439486 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-t7sqv" Oct 30 18:00:31.449834 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.449807 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-td6vr" Oct 30 18:00:31.451173 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:31.451145 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e5fca56_5e6b_404d_84b2_4a876f5034cd.slice/crio-a9dc368ba69895cdc26db91f28d278559f01fa3b3fbf328629be8e9be071331f WatchSource:0}: Error finding container a9dc368ba69895cdc26db91f28d278559f01fa3b3fbf328629be8e9be071331f: Status 404 returned error can't find the container with id a9dc368ba69895cdc26db91f28d278559f01fa3b3fbf328629be8e9be071331f Oct 30 18:00:31.459420 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:31.459383 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d711c2d_0597_4d83_a884_83e50d98378c.slice/crio-29d1f2e2c17db36f7f56db2256ddbe2cae8e4cbc5dfce83649dae19e0c121e49 WatchSource:0}: Error finding container 29d1f2e2c17db36f7f56db2256ddbe2cae8e4cbc5dfce83649dae19e0c121e49: Status 404 returned error can't find the container with id 29d1f2e2c17db36f7f56db2256ddbe2cae8e4cbc5dfce83649dae19e0c121e49 Oct 30 18:00:31.461625 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.461604 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zjlpz" Oct 30 18:00:31.465303 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.465286 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-h6lp9" Oct 30 18:00:31.479891 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:31.479869 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40dd1611_7ec9_411f_920d_e79df7273b90.slice/crio-9c7787af32f326b1a33fc8969a4fdee0caa75dfa0a53db231bc1d4dc60e524db WatchSource:0}: Error finding container 9c7787af32f326b1a33fc8969a4fdee0caa75dfa0a53db231bc1d4dc60e524db: Status 404 returned error can't find the container with id 9c7787af32f326b1a33fc8969a4fdee0caa75dfa0a53db231bc1d4dc60e524db Oct 30 18:00:31.503014 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.502810 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:31.504358 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.504328 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-9bq58" Oct 30 18:00:31.509024 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.509003 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:00:31.511692 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.511659 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:00:31.513879 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:31.513849 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d458e1b_228c_4e8e_b66f_9688583b39db.slice/crio-ffa54663ed165a1dbaec40fa9a7534778162d5b3fe9ee8e0ec42583267dbec8e WatchSource:0}: Error finding container ffa54663ed165a1dbaec40fa9a7534778162d5b3fe9ee8e0ec42583267dbec8e: Status 404 returned error can't find the container with id ffa54663ed165a1dbaec40fa9a7534778162d5b3fe9ee8e0ec42583267dbec8e Oct 30 18:00:31.517055 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.517038 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:00:31.524236 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:31.524210 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4627766d_20a1_46cd_baf9_cff7cc120679.slice/crio-b1a5bdb79ebf602be71a30fb617e07ad2b711b14427053b18a752e2bfd0a19ed WatchSource:0}: Error finding container b1a5bdb79ebf602be71a30fb617e07ad2b711b14427053b18a752e2bfd0a19ed: Status 404 returned error can't find the container with id b1a5bdb79ebf602be71a30fb617e07ad2b711b14427053b18a752e2bfd0a19ed Oct 30 18:00:31.530469 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:31.530442 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76146060_cfc6_4765_82a3_5c5c7601dfda.slice/crio-76c3a34240579e08992e09a4a243257132328a379a721f906a6e74aa401a2783 WatchSource:0}: Error finding container 76c3a34240579e08992e09a4a243257132328a379a721f906a6e74aa401a2783: Status 404 returned error can't find the container with id 76c3a34240579e08992e09a4a243257132328a379a721f906a6e74aa401a2783 Oct 30 18:00:31.534062 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:00:31.534025 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54e2e6ce_5ca5_4de1_b5f6_5a62ad67f3cc.slice/crio-bf6f41d2257c25cec09e37516e4d3dee5a780bc4239b38af7ec4540e8b434c4e WatchSource:0}: Error finding container bf6f41d2257c25cec09e37516e4d3dee5a780bc4239b38af7ec4540e8b434c4e: Status 404 returned error can't find the container with id bf6f41d2257c25cec09e37516e4d3dee5a780bc4239b38af7ec4540e8b434c4e Oct 30 18:00:31.573738 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.573709 2577 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:31.614719 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.614653 2577 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:00:31.914387 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.914167 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xlp\" (UniqueName: \"kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp\") pod \"network-check-target-94r4x\" (UID: \"a3c3fe6d-7fab-4cea-a04e-693b985ceb49\") " pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:31.914501 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.914416 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs\") pod \"network-metrics-daemon-g8h9t\" (UID: \"914c8c09-aa3d-4543-9d91-fa443a09b1ad\") " pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:31.914501 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:31.914453 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:31.914658 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.914600 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:31.914714 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.914662 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret podName:b0df4b9d-89ce-4bb4-b759-68f1b499b52a nodeName:}" failed. No retries permitted until 2026-10-30 18:00:32.914643186 +0000 UTC m=+4.173501945 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret") pod "global-pull-secret-syncer-brgpf" (UID: "b0df4b9d-89ce-4bb4-b759-68f1b499b52a") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:31.914773 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.914745 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:00:31.914773 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.914759 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:00:31.914773 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.914771 2577 projected.go:196] Error preparing data for projected volume kube-api-access-75xlp for pod openshift-network-diagnostics/network-check-target-94r4x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:31.914921 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.914806 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp podName:a3c3fe6d-7fab-4cea-a04e-693b985ceb49 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:32.914795272 +0000 UTC m=+4.173654030 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-75xlp" (UniqueName: "kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp") pod "network-check-target-94r4x" (UID: "a3c3fe6d-7fab-4cea-a04e-693b985ceb49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:31.914921 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.914861 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:31.914921 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:31.914889 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs podName:914c8c09-aa3d-4543-9d91-fa443a09b1ad nodeName:}" failed. No retries permitted until 2026-10-30 18:00:32.914879238 +0000 UTC m=+4.173737986 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs") pod "network-metrics-daemon-g8h9t" (UID: "914c8c09-aa3d-4543-9d91-fa443a09b1ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:32.170812 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:32.170671 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:55:31 +0000 UTC" deadline="2028-08-14 16:48:08.38100895 +0000 UTC" Oct 30 18:00:32.170812 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:32.170727 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15694h47m36.210286234s" Oct 30 18:00:32.237388 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:32.237355 2577 generic.go:356] "Generic (PLEG): container finished" podID="2ace45daccc39fb3be3eb84e89404869" containerID="0540e17ef165c43ba26e209eba50a0d1c6ea585af736bd92cade2103b1424970" exitCode=0 Oct 30 18:00:32.238001 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:32.237663 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" event={"ID":"2ace45daccc39fb3be3eb84e89404869","Type":"ContainerDied","Data":"0540e17ef165c43ba26e209eba50a0d1c6ea585af736bd92cade2103b1424970"} Oct 30 18:00:32.247145 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:32.247118 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k8684" event={"ID":"76146060-cfc6-4765-82a3-5c5c7601dfda","Type":"ContainerStarted","Data":"76c3a34240579e08992e09a4a243257132328a379a721f906a6e74aa401a2783"} Oct 30 18:00:32.252635 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:32.252610 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9bq58" event={"ID":"8754b620-b515-4ee4-9055-469a81126c9e","Type":"ContainerStarted","Data":"f088090189782932f4518a0624192c82cea30262310c71044c8227e22c76e8a5"} Oct 30 18:00:32.266739 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:32.266714 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-td6vr" event={"ID":"8d711c2d-0597-4d83-a884-83e50d98378c","Type":"ContainerStarted","Data":"29d1f2e2c17db36f7f56db2256ddbe2cae8e4cbc5dfce83649dae19e0c121e49"} Oct 30 18:00:32.282114 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:32.282084 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t7sqv" event={"ID":"5e5fca56-5e6b-404d-84b2-4a876f5034cd","Type":"ContainerStarted","Data":"a9dc368ba69895cdc26db91f28d278559f01fa3b3fbf328629be8e9be071331f"} Oct 30 18:00:32.297712 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:32.297680 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" event={"ID":"1bdd65ea12e9d18ae4c0eda6db4fb016","Type":"ContainerStarted","Data":"107bd39ba7e2b3ea22fb40adf5d4235c8c7e9ae6ba148145313783c0f0558545"} Oct 30 18:00:32.302778 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:32.302641 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw76k" event={"ID":"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc","Type":"ContainerStarted","Data":"bf6f41d2257c25cec09e37516e4d3dee5a780bc4239b38af7ec4540e8b434c4e"} Oct 30 18:00:32.308262 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:32.308229 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" event={"ID":"4627766d-20a1-46cd-baf9-cff7cc120679","Type":"ContainerStarted","Data":"b1a5bdb79ebf602be71a30fb617e07ad2b711b14427053b18a752e2bfd0a19ed"} Oct 30 18:00:32.335341 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:32.335308 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" event={"ID":"6d458e1b-228c-4e8e-b66f-9688583b39db","Type":"ContainerStarted","Data":"ffa54663ed165a1dbaec40fa9a7534778162d5b3fe9ee8e0ec42583267dbec8e"} Oct 30 18:00:32.371966 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:32.371925 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h6lp9" event={"ID":"11311f90-d76a-42b5-b6a2-35c0b62bd222","Type":"ContainerStarted","Data":"b0259f4e815dae1a810bd384132879d3c437f9ee8cca15cdf54a11e49ce033e9"} Oct 30 18:00:32.391666 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:32.391634 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zjlpz" event={"ID":"40dd1611-7ec9-411f-920d-e79df7273b90","Type":"ContainerStarted","Data":"9c7787af32f326b1a33fc8969a4fdee0caa75dfa0a53db231bc1d4dc60e524db"} Oct 30 18:00:32.936167 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:32.936131 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xlp\" (UniqueName: \"kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp\") pod \"network-check-target-94r4x\" (UID: \"a3c3fe6d-7fab-4cea-a04e-693b985ceb49\") " pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:32.936346 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:32.936189 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs\") pod \"network-metrics-daemon-g8h9t\" (UID: \"914c8c09-aa3d-4543-9d91-fa443a09b1ad\") " pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:32.936346 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:32.936221 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:32.936443 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:32.936358 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:32.936443 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:32.936421 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret podName:b0df4b9d-89ce-4bb4-b759-68f1b499b52a nodeName:}" failed. No retries permitted until 2026-10-30 18:00:34.936403555 +0000 UTC m=+6.195262302 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret") pod "global-pull-secret-syncer-brgpf" (UID: "b0df4b9d-89ce-4bb4-b759-68f1b499b52a") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:32.936854 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:32.936835 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:00:32.936921 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:32.936866 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:00:32.936921 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:32.936878 2577 projected.go:196] Error preparing data for projected volume kube-api-access-75xlp for pod openshift-network-diagnostics/network-check-target-94r4x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:32.937016 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:32.936924 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp podName:a3c3fe6d-7fab-4cea-a04e-693b985ceb49 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:34.936907653 +0000 UTC m=+6.195766412 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-75xlp" (UniqueName: "kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp") pod "network-check-target-94r4x" (UID: "a3c3fe6d-7fab-4cea-a04e-693b985ceb49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:32.937016 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:32.936985 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:32.937016 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:32.937017 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs podName:914c8c09-aa3d-4543-9d91-fa443a09b1ad nodeName:}" failed. No retries permitted until 2026-10-30 18:00:34.93700568 +0000 UTC m=+6.195864430 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs") pod "network-metrics-daemon-g8h9t" (UID: "914c8c09-aa3d-4543-9d91-fa443a09b1ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:33.171552 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:33.171482 2577 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:55:31 +0000 UTC" deadline="2028-06-21 05:53:13.302867803 +0000 UTC" Oct 30 18:00:33.171552 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:33.171551 2577 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14387h52m40.13132197s" Oct 30 18:00:33.223654 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:33.222800 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:33.223654 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:33.222926 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:00:33.223654 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:33.223296 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:33.223654 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:33.223384 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:00:33.223654 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:33.223452 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:33.223654 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:33.223568 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:00:33.425449 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:33.425392 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" event={"ID":"2ace45daccc39fb3be3eb84e89404869","Type":"ContainerStarted","Data":"80c9a70c4019113c14738128c5a4182c8d7972af6c3d3df16972cf3c0276c9b3"} Oct 30 18:00:33.468768 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:33.468624 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" podStartSLOduration=3.468603269 podStartE2EDuration="3.468603269s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:32.356083952 +0000 UTC m=+3.614942719" watchObservedRunningTime="2026-10-30 18:00:33.468603269 +0000 UTC m=+4.727462030" Oct 30 18:00:34.433788 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:34.433748 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw76k" event={"ID":"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc","Type":"ContainerStarted","Data":"42b94f51956653b1208d881b8325bf23a1bb7c05bd264ede76db1ee61a89c145"} Oct 30 18:00:34.509672 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:34.509615 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" podStartSLOduration=4.509596153 podStartE2EDuration="4.509596153s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:33.469672788 +0000 UTC m=+4.728531555" watchObservedRunningTime="2026-10-30 18:00:34.509596153 +0000 UTC m=+5.768454920" Oct 30 18:00:34.953235 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:34.953186 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xlp\" (UniqueName: \"kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp\") pod \"network-check-target-94r4x\" (UID: \"a3c3fe6d-7fab-4cea-a04e-693b985ceb49\") " pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:34.953408 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:34.953245 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs\") pod \"network-metrics-daemon-g8h9t\" (UID: \"914c8c09-aa3d-4543-9d91-fa443a09b1ad\") " pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:34.953408 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:34.953278 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:34.953550 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:34.953417 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:34.953550 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:34.953476 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret podName:b0df4b9d-89ce-4bb4-b759-68f1b499b52a nodeName:}" failed. No retries permitted until 2026-10-30 18:00:38.953458614 +0000 UTC m=+10.212317373 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret") pod "global-pull-secret-syncer-brgpf" (UID: "b0df4b9d-89ce-4bb4-b759-68f1b499b52a") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:34.953930 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:34.953904 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:00:34.953930 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:34.953933 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:00:34.954078 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:34.953946 2577 projected.go:196] Error preparing data for projected volume kube-api-access-75xlp for pod openshift-network-diagnostics/network-check-target-94r4x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:34.954078 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:34.953990 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp podName:a3c3fe6d-7fab-4cea-a04e-693b985ceb49 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:38.953976666 +0000 UTC m=+10.212835416 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-75xlp" (UniqueName: "kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp") pod "network-check-target-94r4x" (UID: "a3c3fe6d-7fab-4cea-a04e-693b985ceb49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:34.954078 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:34.954051 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:34.954078 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:34.954081 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs podName:914c8c09-aa3d-4543-9d91-fa443a09b1ad nodeName:}" failed. No retries permitted until 2026-10-30 18:00:38.954070988 +0000 UTC m=+10.212929739 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs") pod "network-metrics-daemon-g8h9t" (UID: "914c8c09-aa3d-4543-9d91-fa443a09b1ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:35.222630 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:35.222532 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:35.222771 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:35.222670 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:00:35.222771 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:35.222743 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:35.222878 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:35.222810 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:00:35.222878 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:35.222856 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:35.222980 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:35.222916 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:00:35.446104 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:35.445310 2577 generic.go:356] "Generic (PLEG): container finished" podID="54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc" containerID="42b94f51956653b1208d881b8325bf23a1bb7c05bd264ede76db1ee61a89c145" exitCode=0 Oct 30 18:00:35.446104 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:35.445368 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw76k" event={"ID":"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc","Type":"ContainerDied","Data":"42b94f51956653b1208d881b8325bf23a1bb7c05bd264ede76db1ee61a89c145"} Oct 30 18:00:36.453313 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:36.452687 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zjlpz" event={"ID":"40dd1611-7ec9-411f-920d-e79df7273b90","Type":"ContainerStarted","Data":"dcc072cf11aecfb17aa50e1669f1f0b18377a70ba03b02af82e26ff51d674981"} Oct 30 18:00:37.223316 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:37.223210 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:37.223463 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:37.223348 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:00:37.223701 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:37.223677 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:37.224093 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:37.224065 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:37.229707 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:37.229674 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:00:37.229825 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:37.229806 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:00:37.457228 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:37.457191 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-h6lp9" event={"ID":"11311f90-d76a-42b5-b6a2-35c0b62bd222","Type":"ContainerStarted","Data":"20efa28376b0fa344902943287bd0cf68690866da5ac50548dc02ff04965f6fa"} Oct 30 18:00:37.460063 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:37.460035 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" event={"ID":"4627766d-20a1-46cd-baf9-cff7cc120679","Type":"ContainerStarted","Data":"eec0a378d41083a52650a8e19ad86928cdf4144abdc3f808e56dec1af1487113"} Oct 30 18:00:37.503094 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:37.502992 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zjlpz" podStartSLOduration=4.72269717 podStartE2EDuration="8.502974298s" totalImagesPullingTime="3.780277128s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:36.495224423 +0000 UTC m=+7.754083190" watchObservedRunningTime="2026-10-30 18:00:37.502974298 +0000 UTC m=+8.761833065" Oct 30 18:00:38.663133 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:38.663034 2577 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 18:00:38.987213 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:38.986673 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xlp\" (UniqueName: \"kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp\") pod \"network-check-target-94r4x\" (UID: \"a3c3fe6d-7fab-4cea-a04e-693b985ceb49\") " pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:38.987213 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:38.986724 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs\") pod \"network-metrics-daemon-g8h9t\" (UID: \"914c8c09-aa3d-4543-9d91-fa443a09b1ad\") " pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:38.987213 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:38.986759 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:38.987213 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:38.986885 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:38.987213 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:38.986944 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret podName:b0df4b9d-89ce-4bb4-b759-68f1b499b52a nodeName:}" failed. No retries permitted until 2026-10-30 18:00:46.986927851 +0000 UTC m=+18.245786598 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret") pod "global-pull-secret-syncer-brgpf" (UID: "b0df4b9d-89ce-4bb4-b759-68f1b499b52a") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:38.987213 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:38.987027 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:00:38.987213 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:38.987044 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:00:38.987213 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:38.987056 2577 projected.go:196] Error preparing data for projected volume kube-api-access-75xlp for pod openshift-network-diagnostics/network-check-target-94r4x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:38.987213 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:38.987090 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp podName:a3c3fe6d-7fab-4cea-a04e-693b985ceb49 nodeName:}" failed. No retries permitted until 2026-10-30 18:00:46.987078589 +0000 UTC m=+18.245937335 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-75xlp" (UniqueName: "kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp") pod "network-check-target-94r4x" (UID: "a3c3fe6d-7fab-4cea-a04e-693b985ceb49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:38.987213 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:38.987150 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:38.987213 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:38.987189 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs podName:914c8c09-aa3d-4543-9d91-fa443a09b1ad nodeName:}" failed. No retries permitted until 2026-10-30 18:00:46.987169479 +0000 UTC m=+18.246028226 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs") pod "network-metrics-daemon-g8h9t" (UID: "914c8c09-aa3d-4543-9d91-fa443a09b1ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:39.173159 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:39.173077 2577 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:00:38.663054867Z","UUID":"49bbca63-bf9e-460f-b510-b77e52e89a87","Handler":null,"Name":"","Endpoint":""} Oct 30 18:00:39.174746 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:39.174722 2577 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 18:00:39.174844 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:39.174754 2577 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 18:00:39.221148 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:39.221124 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:39.221275 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:39.221212 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:00:39.221275 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:39.221252 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:39.221386 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:39.221290 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:00:39.221386 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:39.221343 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:39.221482 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:39.221406 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:00:39.468255 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:39.468070 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-td6vr" event={"ID":"8d711c2d-0597-4d83-a884-83e50d98378c","Type":"ContainerStarted","Data":"7d9a9ba8ec9906a7185a71e02c99aa8bc1101cc2d71531cf39df9680be711861"} Oct 30 18:00:39.484632 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:39.484548 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" event={"ID":"4627766d-20a1-46cd-baf9-cff7cc120679","Type":"ContainerStarted","Data":"f13948af77f28b652eebfe6c9fb90f429922c9c8e08a9fef5e6f36b58c62eb47"} Oct 30 18:00:39.487665 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:39.487626 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-9bq58" event={"ID":"8754b620-b515-4ee4-9055-469a81126c9e","Type":"ContainerStarted","Data":"0d3da71b7367e559c02cda96b50c3a64fbaf35bf350de4675f12e3ff60033c87"} Oct 30 18:00:39.510901 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:39.510859 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-td6vr" podStartSLOduration=2.9973000560000003 podStartE2EDuration="10.51084638s" totalImagesPullingTime="7.513546324s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:39.510436624 +0000 UTC m=+10.769295390" watchObservedRunningTime="2026-10-30 18:00:39.51084638 +0000 UTC m=+10.769705146" Oct 30 18:00:39.511091 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:39.511068 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-h6lp9" podStartSLOduration=5.560812272 podStartE2EDuration="10.51106118s" totalImagesPullingTime="4.950248908s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:37.503267912 +0000 UTC m=+8.762126679" watchObservedRunningTime="2026-10-30 18:00:39.51106118 +0000 UTC m=+10.769919943" Oct 30 18:00:39.567125 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:39.567065 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-9bq58" podStartSLOduration=3.059199449 podStartE2EDuration="10.567051687s" totalImagesPullingTime="7.507852238s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:39.56635723 +0000 UTC m=+10.825215999" watchObservedRunningTime="2026-10-30 18:00:39.567051687 +0000 UTC m=+10.825910453" Oct 30 18:00:40.492175 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:40.492137 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" event={"ID":"4627766d-20a1-46cd-baf9-cff7cc120679","Type":"ContainerStarted","Data":"71b41657ae11d5e0eb1bd1b74c6f38059beb067f0fef12fb8320d630aac3fbca"} Oct 30 18:00:40.496083 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:40.494669 2577 generic.go:356] "Generic (PLEG): container finished" podID="54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc" containerID="6308d0b4b938a0996825585cfde44cf56e5f4f8466d38372185a14fda6c78063" exitCode=0 Oct 30 18:00:40.496083 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:40.495615 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw76k" event={"ID":"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc","Type":"ContainerDied","Data":"6308d0b4b938a0996825585cfde44cf56e5f4f8466d38372185a14fda6c78063"} Oct 30 18:00:40.536430 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:40.536381 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" podStartSLOduration=3.708907362 podStartE2EDuration="11.536362659s" totalImagesPullingTime="7.827455297s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:40.535852244 +0000 UTC m=+11.794711011" watchObservedRunningTime="2026-10-30 18:00:40.536362659 +0000 UTC m=+11.795221424" Oct 30 18:00:40.690437 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:40.690401 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zjlpz" Oct 30 18:00:40.691194 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:40.691172 2577 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zjlpz" Oct 30 18:00:41.221149 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:41.221115 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:41.221149 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:41.221137 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:41.221378 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:41.221125 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:41.221378 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:41.221229 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:00:41.221378 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:41.221289 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:00:41.221571 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:41.221397 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:00:41.498542 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:41.498424 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-t7sqv" event={"ID":"5e5fca56-5e6b-404d-84b2-4a876f5034cd","Type":"ContainerStarted","Data":"9a51d72984c13e9c4c7e0f6661c389b6fa1c9d9a26bb06741d0d1c711454346b"} Oct 30 18:00:41.539836 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:41.539739 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-t7sqv" podStartSLOduration=4.785854183 podStartE2EDuration="12.539719246s" totalImagesPullingTime="7.753865063s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:41.539251036 +0000 UTC m=+12.798109804" watchObservedRunningTime="2026-10-30 18:00:41.539719246 +0000 UTC m=+12.798578037" Oct 30 18:00:42.501877 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:42.501840 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-k8684" event={"ID":"76146060-cfc6-4765-82a3-5c5c7601dfda","Type":"ContainerStarted","Data":"75b2819eb3fd96d763a6c980b58b479d7dc4a0312c22abb4d716f43dd8aad94c"} Oct 30 18:00:42.504566 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:42.504536 2577 generic.go:356] "Generic (PLEG): container finished" podID="54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc" containerID="588ea41a3006a7398c9a8fec71a1f1c520724c34485b355bb719aa79a10672d4" exitCode=0 Oct 30 18:00:42.504691 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:42.504612 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw76k" event={"ID":"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc","Type":"ContainerDied","Data":"588ea41a3006a7398c9a8fec71a1f1c520724c34485b355bb719aa79a10672d4"} Oct 30 18:00:42.545264 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:42.545217 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-k8684" podStartSLOduration=3.413510737 podStartE2EDuration="13.545203863s" totalImagesPullingTime="10.131693126s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:42.545000917 +0000 UTC m=+13.803859684" watchObservedRunningTime="2026-10-30 18:00:42.545203863 +0000 UTC m=+13.804062629" Oct 30 18:00:43.220731 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:43.220707 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:43.220731 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:43.220722 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:43.220887 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:43.220837 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:00:43.220975 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:43.220925 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:43.221043 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:43.221018 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:00:43.221144 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:43.221104 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:00:43.509845 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:43.509535 2577 generic.go:356] "Generic (PLEG): container finished" podID="54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc" containerID="13b762ec051cb6abf2b396b35e54a7a98e917f1725d472c8009ea76cfb6634eb" exitCode=0 Oct 30 18:00:43.509845 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:43.509584 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw76k" event={"ID":"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc","Type":"ContainerDied","Data":"13b762ec051cb6abf2b396b35e54a7a98e917f1725d472c8009ea76cfb6634eb"} Oct 30 18:00:44.683540 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:44.683482 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zjlpz" Oct 30 18:00:44.684009 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:44.683689 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:00:44.684807 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:44.684652 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zjlpz" Oct 30 18:00:45.220609 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:45.220586 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:45.220609 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:45.220605 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:45.220867 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:45.220698 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:00:45.220867 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:45.220736 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:00:45.220867 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:45.220789 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:45.220990 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:45.220892 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:00:45.516993 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:45.516929 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw76k" event={"ID":"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc","Type":"ContainerStarted","Data":"b70c97d6538689d0db1b7d30e1730076599dda21a5702e1639cef84c2f53b41f"} Oct 30 18:00:46.522441 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:46.522228 2577 generic.go:356] "Generic (PLEG): container finished" podID="54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc" containerID="b70c97d6538689d0db1b7d30e1730076599dda21a5702e1639cef84c2f53b41f" exitCode=0 Oct 30 18:00:46.522441 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:46.522309 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw76k" event={"ID":"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc","Type":"ContainerDied","Data":"b70c97d6538689d0db1b7d30e1730076599dda21a5702e1639cef84c2f53b41f"} Oct 30 18:00:46.525766 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:46.525738 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" event={"ID":"6d458e1b-228c-4e8e-b66f-9688583b39db","Type":"ContainerStarted","Data":"c8f52f0a8474d327d8e8288c5e0b7a90365d9712ad21eaa90bda749274d93c34"} Oct 30 18:00:46.525856 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:46.525829 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" event={"ID":"6d458e1b-228c-4e8e-b66f-9688583b39db","Type":"ContainerStarted","Data":"7bcf057a3bf3a5a80a9f46b3708cbfa6c3836fe95929d98d546e7895f3a28633"} Oct 30 18:00:46.525905 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:46.525880 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" event={"ID":"6d458e1b-228c-4e8e-b66f-9688583b39db","Type":"ContainerStarted","Data":"4de5d4f5b5f4c9a33b8cb90e7368ef3223479c500caa5b25212f6dde1689f13b"} Oct 30 18:00:46.525935 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:46.525927 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" event={"ID":"6d458e1b-228c-4e8e-b66f-9688583b39db","Type":"ContainerStarted","Data":"ca37bec38b130962f993282abc2662908231c3fa65811ba7b8e94ea490e5a5d0"} Oct 30 18:00:46.525984 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:46.525972 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" event={"ID":"6d458e1b-228c-4e8e-b66f-9688583b39db","Type":"ContainerStarted","Data":"989f6e2e9f9b8b995de95e9645702d0abd6b1b1e039e33a201f9b12a94a902c4"} Oct 30 18:00:46.526029 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:46.526018 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" event={"ID":"6d458e1b-228c-4e8e-b66f-9688583b39db","Type":"ContainerStarted","Data":"fcd1f9cc96601a47664b5cef2426b833fcf0ecd946ff26842ccf7e377861d596"} Oct 30 18:00:47.049336 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:47.049306 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xlp\" (UniqueName: \"kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp\") pod \"network-check-target-94r4x\" (UID: \"a3c3fe6d-7fab-4cea-a04e-693b985ceb49\") " pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:47.049336 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:47.049340 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs\") pod \"network-metrics-daemon-g8h9t\" (UID: \"914c8c09-aa3d-4543-9d91-fa443a09b1ad\") " pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:47.049581 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:47.049387 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:47.049581 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:47.049470 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:47.049581 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:47.049471 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:00:47.049581 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:47.049496 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:00:47.049581 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:47.049503 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:47.049581 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:47.049526 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret podName:b0df4b9d-89ce-4bb4-b759-68f1b499b52a nodeName:}" failed. No retries permitted until 2026-10-30 18:01:03.049502147 +0000 UTC m=+34.308360891 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret") pod "global-pull-secret-syncer-brgpf" (UID: "b0df4b9d-89ce-4bb4-b759-68f1b499b52a") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:00:47.049581 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:47.049535 2577 projected.go:196] Error preparing data for projected volume kube-api-access-75xlp for pod openshift-network-diagnostics/network-check-target-94r4x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:47.049581 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:47.049572 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs podName:914c8c09-aa3d-4543-9d91-fa443a09b1ad nodeName:}" failed. No retries permitted until 2026-10-30 18:01:03.049559781 +0000 UTC m=+34.308418525 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs") pod "network-metrics-daemon-g8h9t" (UID: "914c8c09-aa3d-4543-9d91-fa443a09b1ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:00:47.049581 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:47.049585 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp podName:a3c3fe6d-7fab-4cea-a04e-693b985ceb49 nodeName:}" failed. No retries permitted until 2026-10-30 18:01:03.049578936 +0000 UTC m=+34.308437679 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-75xlp" (UniqueName: "kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp") pod "network-check-target-94r4x" (UID: "a3c3fe6d-7fab-4cea-a04e-693b985ceb49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:00:47.223608 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:47.223584 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:47.223741 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:47.223676 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:47.223741 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:47.223678 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:00:47.223819 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:47.223678 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:47.223819 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:47.223747 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:00:47.223893 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:47.223822 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:00:47.531085 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:47.531007 2577 generic.go:356] "Generic (PLEG): container finished" podID="54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc" containerID="a62e035b8e8571735eec3afbc03b6d8b970d1a2cd2ada35798350403d5c6a40b" exitCode=0 Oct 30 18:00:47.531085 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:47.531076 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw76k" event={"ID":"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc","Type":"ContainerDied","Data":"a62e035b8e8571735eec3afbc03b6d8b970d1a2cd2ada35798350403d5c6a40b"} Oct 30 18:00:48.536787 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:48.536755 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-mw76k" event={"ID":"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc","Type":"ContainerStarted","Data":"ea5e9cbba92fa50faa6585ea06184483102e49473e16fdaae319b189b1a9f3c5"} Oct 30 18:00:48.542440 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:48.542416 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" event={"ID":"6d458e1b-228c-4e8e-b66f-9688583b39db","Type":"ContainerStarted","Data":"4926d89627eec2614edfb08aadab7a675d4f753ecfba8b3c8fb8fdc3d552222b"} Oct 30 18:00:49.221464 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:49.221429 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:49.221640 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:49.221543 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:00:49.221640 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:49.221626 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:49.221764 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:49.221708 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:00:49.221764 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:49.221756 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:49.221873 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:49.221825 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:00:50.550976 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:50.550942 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" event={"ID":"6d458e1b-228c-4e8e-b66f-9688583b39db","Type":"ContainerStarted","Data":"add831aa21b061d1f0c15859329f29c4406f7f818c309985ef2fc50f291f5772"} Oct 30 18:00:50.551596 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:50.551194 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:50.551596 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:50.551270 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:50.568410 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:50.568250 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:50.568584 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:50.568555 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:50.644377 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:50.644328 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-mw76k" podStartSLOduration=11.545619896 podStartE2EDuration="21.644313743s" totalImagesPullingTime="10.098693847s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:48.596604817 +0000 UTC m=+19.855463584" watchObservedRunningTime="2026-10-30 18:00:50.644313743 +0000 UTC m=+21.903172505" Oct 30 18:00:50.644601 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:50.644577 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" podStartSLOduration=7.167073418 podStartE2EDuration="21.644568803s" totalImagesPullingTime="14.477495385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:00:50.630552389 +0000 UTC m=+21.889411155" watchObservedRunningTime="2026-10-30 18:00:50.644568803 +0000 UTC m=+21.903427570" Oct 30 18:00:51.000769 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:51.000739 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-td6vr_8d711c2d-0597-4d83-a884-83e50d98378c/dns-node-resolver/0.log" Oct 30 18:00:51.223890 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:51.223860 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:51.224041 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:51.223935 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:51.224041 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:51.224019 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:00:51.224041 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:51.224027 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:51.224199 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:51.224110 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:00:51.224248 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:51.224206 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:00:51.549659 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:51.549638 2577 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h6lp9_11311f90-d76a-42b5-b6a2-35c0b62bd222/node-ca/0.log" Oct 30 18:00:51.553159 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:51.553139 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:00:52.554603 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:52.554575 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:00:53.220982 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:53.220958 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:53.220982 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:53.220977 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:53.221164 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:53.220960 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:53.221164 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:53.221061 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:00:53.221164 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:53.221122 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:00:53.221250 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:53.221214 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:00:55.223581 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:55.223548 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:55.223969 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:55.223604 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:55.223969 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:55.223630 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:55.223969 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:55.223708 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:00:55.223969 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:55.223767 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:00:55.223969 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:55.223829 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:00:57.223812 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:57.223776 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:57.223812 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:57.223800 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:57.224333 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:57.223780 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:57.224333 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:57.223864 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:00:57.224333 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:57.223926 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:00:57.224333 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:57.223985 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:00:58.751777 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:58.751747 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:00:58.752170 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:58.751920 2577 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:00:58.762468 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:58.762435 2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" podUID="6d458e1b-228c-4e8e-b66f-9688583b39db" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 18:00:58.770945 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:58.770920 2577 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" podUID="6d458e1b-228c-4e8e-b66f-9688583b39db" containerName="ovnkube-controller" probeResult="failure" output="" Oct 30 18:00:59.222602 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:59.222498 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:00:59.222602 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:59.222555 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:00:59.222765 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:00:59.222657 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:00:59.222765 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:59.222674 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:00:59.222827 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:59.222772 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:00:59.222879 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:00:59.222860 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:01:01.220497 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:01.220469 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:01.220902 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:01.220469 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:01:01.220902 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:01.220595 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:01:01.220902 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:01.220659 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:01:01.220902 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:01.220694 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:01:01.220902 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:01.220752 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:01:03.072736 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:03.072704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xlp\" (UniqueName: \"kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp\") pod \"network-check-target-94r4x\" (UID: \"a3c3fe6d-7fab-4cea-a04e-693b985ceb49\") " pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:01:03.072736 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:03.072743 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs\") pod \"network-metrics-daemon-g8h9t\" (UID: \"914c8c09-aa3d-4543-9d91-fa443a09b1ad\") " pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:01:03.073156 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:03.072764 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:03.073156 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:03.072861 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 18:01:03.073156 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:03.072863 2577 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:01:03.073156 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:03.072877 2577 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 18:01:03.073156 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:03.072895 2577 projected.go:196] Error preparing data for projected volume kube-api-access-75xlp for pod openshift-network-diagnostics/network-check-target-94r4x: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:01:03.073156 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:03.072863 2577 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 18:01:03.073156 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:03.072927 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs podName:914c8c09-aa3d-4543-9d91-fa443a09b1ad nodeName:}" failed. No retries permitted until 2026-10-30 18:01:35.072909882 +0000 UTC m=+66.331768647 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs") pod "network-metrics-daemon-g8h9t" (UID: "914c8c09-aa3d-4543-9d91-fa443a09b1ad") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 18:01:03.073156 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:03.072944 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp podName:a3c3fe6d-7fab-4cea-a04e-693b985ceb49 nodeName:}" failed. No retries permitted until 2026-10-30 18:01:35.072936678 +0000 UTC m=+66.331795435 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-75xlp" (UniqueName: "kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp") pod "network-check-target-94r4x" (UID: "a3c3fe6d-7fab-4cea-a04e-693b985ceb49") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 18:01:03.073156 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:03.072955 2577 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret podName:b0df4b9d-89ce-4bb4-b759-68f1b499b52a nodeName:}" failed. No retries permitted until 2026-10-30 18:01:35.072950099 +0000 UTC m=+66.331808843 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret") pod "global-pull-secret-syncer-brgpf" (UID: "b0df4b9d-89ce-4bb4-b759-68f1b499b52a") : object "kube-system"/"original-pull-secret" not registered Oct 30 18:01:03.220599 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:03.220574 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:01:03.220599 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:03.220601 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:01:03.220771 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:03.220610 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:03.220771 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:03.220682 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:01:03.220834 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:03.220760 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:01:03.220867 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:03.220842 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:01:05.223666 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:05.223634 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:01:05.223666 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:05.223665 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:01:05.224133 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:05.223730 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:01:05.224133 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:05.223753 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:05.224133 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:05.223823 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:01:05.224133 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:05.223906 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:01:07.220681 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:07.220649 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:01:07.220681 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:07.220683 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:07.221069 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:07.220656 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:01:07.221069 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:07.220771 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:01:07.221069 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:07.220845 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:01:07.221069 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:07.220896 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:01:09.222761 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:09.222735 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:09.223226 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:09.222828 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:01:09.223226 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:09.222858 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:01:09.223226 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:09.222935 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:01:09.223226 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:09.222965 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:01:09.223226 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:09.223077 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:01:10.496977 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:10.496941 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g8h9t"] Oct 30 18:01:10.497435 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:10.497107 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:01:10.497435 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:10.497204 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:01:10.497667 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:10.497649 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-94r4x"] Oct 30 18:01:10.497747 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:10.497735 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:01:10.497821 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:10.497807 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:01:10.498173 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:10.498153 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-brgpf"] Oct 30 18:01:10.498280 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:10.498262 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:10.498367 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:10.498350 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:01:12.220567 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:12.220386 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:01:12.220943 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:12.220464 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:12.220943 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:12.220730 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:01:12.220943 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:12.220629 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:01:12.220943 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:12.220471 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:01:12.220943 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:12.220828 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:01:14.220621 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:14.220587 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:01:14.220621 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:14.220625 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:14.221086 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:14.220587 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:01:14.221086 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:14.220721 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-g8h9t" podUID="914c8c09-aa3d-4543-9d91-fa443a09b1ad" Oct 30 18:01:14.221086 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:14.220773 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-94r4x" podUID="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" Oct 30 18:01:14.221086 ip-10-0-141-106 kubenswrapper[2577]: E1030 18:01:14.220839 2577 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-brgpf" podUID="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" Oct 30 18:01:15.442607 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.442582 2577 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeReady" Oct 30 18:01:15.442922 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.442723 2577 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 18:01:15.501182 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.501156 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-st6k6"] Oct 30 18:01:15.503039 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.503020 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:15.504682 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.504662 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-b9kd2"] Oct 30 18:01:15.506168 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.506151 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:15.522972 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.522914 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 18:01:15.526558 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.525270 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-kqvrz"] Oct 30 18:01:15.526558 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.525657 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 18:01:15.526558 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.526273 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wc69q\"" Oct 30 18:01:15.527561 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.527540 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-st6k6"] Oct 30 18:01:15.527665 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.527640 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kqvrz" Oct 30 18:01:15.528439 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.528400 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 18:01:15.528558 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.528451 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 18:01:15.536547 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.536528 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b9kd2"] Oct 30 18:01:15.551403 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.551375 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kqvrz"] Oct 30 18:01:15.560590 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.560565 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54bdc818-2b49-4f72-a168-6958864242fd-tmp-dir\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:15.560687 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.560602 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rjmx\" (UniqueName: \"kubernetes.io/projected/7bd897e6-be5e-4d00-8faf-b2544c55a3f6-kube-api-access-4rjmx\") pod \"ingress-canary-kqvrz\" (UID: \"7bd897e6-be5e-4d00-8faf-b2544c55a3f6\") " pod="openshift-ingress-canary/ingress-canary-kqvrz" Oct 30 18:01:15.560687 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.560619 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31af1374-0d44-4636-bb2a-8e3836ac56b2-crio-socket\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:15.560756 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.560722 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54bdc818-2b49-4f72-a168-6958864242fd-config-volume\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:15.560756 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.560749 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bhp2k\" (UniqueName: \"kubernetes.io/projected/31af1374-0d44-4636-bb2a-8e3836ac56b2-kube-api-access-bhp2k\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:15.560812 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.560774 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31af1374-0d44-4636-bb2a-8e3836ac56b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:15.560812 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.560793 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54bdc818-2b49-4f72-a168-6958864242fd-metrics-tls\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:15.560871 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.560846 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bd897e6-be5e-4d00-8faf-b2544c55a3f6-cert\") pod \"ingress-canary-kqvrz\" (UID: \"7bd897e6-be5e-4d00-8faf-b2544c55a3f6\") " pod="openshift-ingress-canary/ingress-canary-kqvrz" Oct 30 18:01:15.560899 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.560868 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31af1374-0d44-4636-bb2a-8e3836ac56b2-data-volume\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:15.560932 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.560921 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31af1374-0d44-4636-bb2a-8e3836ac56b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:15.560965 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.560946 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8clmh\" (UniqueName: \"kubernetes.io/projected/54bdc818-2b49-4f72-a168-6958864242fd-kube-api-access-8clmh\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:15.621892 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.621866 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 18:01:15.621892 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.621895 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qfvjp\"" Oct 30 18:01:15.622090 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.621928 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d6j24\"" Oct 30 18:01:15.622090 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.621970 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 18:01:15.625909 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.625895 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 18:01:15.626008 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.625993 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 18:01:15.626044 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.626032 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 18:01:15.661240 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.661173 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rjmx\" (UniqueName: \"kubernetes.io/projected/7bd897e6-be5e-4d00-8faf-b2544c55a3f6-kube-api-access-4rjmx\") pod \"ingress-canary-kqvrz\" (UID: \"7bd897e6-be5e-4d00-8faf-b2544c55a3f6\") " pod="openshift-ingress-canary/ingress-canary-kqvrz" Oct 30 18:01:15.661240 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.661198 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31af1374-0d44-4636-bb2a-8e3836ac56b2-crio-socket\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:15.661240 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.661225 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54bdc818-2b49-4f72-a168-6958864242fd-config-volume\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:15.661476 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.661243 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhp2k\" (UniqueName: \"kubernetes.io/projected/31af1374-0d44-4636-bb2a-8e3836ac56b2-kube-api-access-bhp2k\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:15.661476 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.661267 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31af1374-0d44-4636-bb2a-8e3836ac56b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:15.661476 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.661281 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54bdc818-2b49-4f72-a168-6958864242fd-metrics-tls\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:15.661476 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.661416 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bd897e6-be5e-4d00-8faf-b2544c55a3f6-cert\") pod \"ingress-canary-kqvrz\" (UID: \"7bd897e6-be5e-4d00-8faf-b2544c55a3f6\") " pod="openshift-ingress-canary/ingress-canary-kqvrz" Oct 30 18:01:15.661476 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.661454 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31af1374-0d44-4636-bb2a-8e3836ac56b2-crio-socket\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:15.661668 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.661456 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31af1374-0d44-4636-bb2a-8e3836ac56b2-data-volume\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:15.661724 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.661704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31af1374-0d44-4636-bb2a-8e3836ac56b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:15.661774 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.661754 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8clmh\" (UniqueName: \"kubernetes.io/projected/54bdc818-2b49-4f72-a168-6958864242fd-kube-api-access-8clmh\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:15.661825 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.661784 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31af1374-0d44-4636-bb2a-8e3836ac56b2-data-volume\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:15.661825 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.661797 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54bdc818-2b49-4f72-a168-6958864242fd-tmp-dir\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:15.662014 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.661960 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54bdc818-2b49-4f72-a168-6958864242fd-config-volume\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:15.662122 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.662074 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54bdc818-2b49-4f72-a168-6958864242fd-tmp-dir\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:15.662223 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.662204 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31af1374-0d44-4636-bb2a-8e3836ac56b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:15.665300 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.665279 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54bdc818-2b49-4f72-a168-6958864242fd-metrics-tls\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:15.665387 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.665276 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31af1374-0d44-4636-bb2a-8e3836ac56b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:15.665432 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.665425 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bd897e6-be5e-4d00-8faf-b2544c55a3f6-cert\") pod \"ingress-canary-kqvrz\" (UID: \"7bd897e6-be5e-4d00-8faf-b2544c55a3f6\") " pod="openshift-ingress-canary/ingress-canary-kqvrz" Oct 30 18:01:15.682633 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.682607 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rjmx\" (UniqueName: \"kubernetes.io/projected/7bd897e6-be5e-4d00-8faf-b2544c55a3f6-kube-api-access-4rjmx\") pod \"ingress-canary-kqvrz\" (UID: \"7bd897e6-be5e-4d00-8faf-b2544c55a3f6\") " pod="openshift-ingress-canary/ingress-canary-kqvrz" Oct 30 18:01:15.682766 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.682748 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8clmh\" (UniqueName: \"kubernetes.io/projected/54bdc818-2b49-4f72-a168-6958864242fd-kube-api-access-8clmh\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:15.683801 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.683779 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhp2k\" (UniqueName: \"kubernetes.io/projected/31af1374-0d44-4636-bb2a-8e3836ac56b2-kube-api-access-bhp2k\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:15.815678 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.815643 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:15.821178 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.821162 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:15.837928 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:15.837897 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-kqvrz" Oct 30 18:01:16.131071 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:16.131045 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-st6k6"] Oct 30 18:01:16.131696 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:16.131675 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-b9kd2"] Oct 30 18:01:16.133174 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:01:16.133144 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54bdc818_2b49_4f72_a168_6958864242fd.slice/crio-9dbc196724d7a247bbf69f1d92235f3cb08b6e2a86590d7a7b9e729289a7ddfc WatchSource:0}: Error finding container 9dbc196724d7a247bbf69f1d92235f3cb08b6e2a86590d7a7b9e729289a7ddfc: Status 404 returned error can't find the container with id 9dbc196724d7a247bbf69f1d92235f3cb08b6e2a86590d7a7b9e729289a7ddfc Oct 30 18:01:16.133427 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:01:16.133402 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31af1374_0d44_4636_bb2a_8e3836ac56b2.slice/crio-5350ba5b6e2fc68a67e5d0f1eb3a7426e97c5a39f651ed171b95e2e062cf719e WatchSource:0}: Error finding container 5350ba5b6e2fc68a67e5d0f1eb3a7426e97c5a39f651ed171b95e2e062cf719e: Status 404 returned error can't find the container with id 5350ba5b6e2fc68a67e5d0f1eb3a7426e97c5a39f651ed171b95e2e062cf719e Oct 30 18:01:16.154609 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:16.154589 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-kqvrz"] Oct 30 18:01:16.166549 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:01:16.166487 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7bd897e6_be5e_4d00_8faf_b2544c55a3f6.slice/crio-03aabceb0285bf62c690b8cbb84759a7f4ceaaca8894efab90b4da611298d336 WatchSource:0}: Error finding container 03aabceb0285bf62c690b8cbb84759a7f4ceaaca8894efab90b4da611298d336: Status 404 returned error can't find the container with id 03aabceb0285bf62c690b8cbb84759a7f4ceaaca8894efab90b4da611298d336 Oct 30 18:01:16.220419 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:16.220395 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:16.220541 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:16.220417 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:01:16.220541 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:16.220428 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:01:16.237049 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:16.237020 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:01:16.237176 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:16.237066 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:01:16.237176 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:16.237101 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qrbw2\"" Oct 30 18:01:16.237264 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:16.237196 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:01:16.237264 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:16.237196 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdst5\"" Oct 30 18:01:16.237264 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:16.237232 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khr2s\"" Oct 30 18:01:16.237354 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:16.237327 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:01:16.608524 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:16.608464 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kqvrz" event={"ID":"7bd897e6-be5e-4d00-8faf-b2544c55a3f6","Type":"ContainerStarted","Data":"03aabceb0285bf62c690b8cbb84759a7f4ceaaca8894efab90b4da611298d336"} Oct 30 18:01:16.611003 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:16.610775 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9kd2" event={"ID":"31af1374-0d44-4636-bb2a-8e3836ac56b2","Type":"ContainerStarted","Data":"ff98c48457eb3f64bedc9da808b7f2147db9691cb456552caa45c733e68a9983"} Oct 30 18:01:16.611003 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:16.610837 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9kd2" event={"ID":"31af1374-0d44-4636-bb2a-8e3836ac56b2","Type":"ContainerStarted","Data":"5350ba5b6e2fc68a67e5d0f1eb3a7426e97c5a39f651ed171b95e2e062cf719e"} Oct 30 18:01:16.612014 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:16.611988 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-st6k6" event={"ID":"54bdc818-2b49-4f72-a168-6958864242fd","Type":"ContainerStarted","Data":"9dbc196724d7a247bbf69f1d92235f3cb08b6e2a86590d7a7b9e729289a7ddfc"} Oct 30 18:01:17.617047 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:17.617002 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9kd2" event={"ID":"31af1374-0d44-4636-bb2a-8e3836ac56b2","Type":"ContainerStarted","Data":"49b608859c2d4498fb2c6a590f071571ed55372a4d556bc99e4d24105456e976"} Oct 30 18:01:18.485236 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.485202 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-5zplp"] Oct 30 18:01:18.487401 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.487377 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.561922 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.561893 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 18:01:18.562305 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.562260 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:01:18.562305 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.562268 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qzhdb\"" Oct 30 18:01:18.562594 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.562571 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:01:18.562701 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.562572 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 18:01:18.562701 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.562670 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 18:01:18.562789 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.562578 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 18:01:18.588321 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.588239 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.588321 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.588284 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/17beb356-34fc-4c4a-97a4-72f00c649e7b-root\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.588321 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.588310 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17beb356-34fc-4c4a-97a4-72f00c649e7b-metrics-client-ca\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.588592 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.588449 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-wtmp\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.588592 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.588493 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-textfile\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.588699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.588585 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.588699 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.588656 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-tls\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.588778 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.588725 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grpvw\" (UniqueName: \"kubernetes.io/projected/17beb356-34fc-4c4a-97a4-72f00c649e7b-kube-api-access-grpvw\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.588816 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.588791 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17beb356-34fc-4c4a-97a4-72f00c649e7b-sys\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.621710 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.621672 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-b9kd2" event={"ID":"31af1374-0d44-4636-bb2a-8e3836ac56b2","Type":"ContainerStarted","Data":"0f4558370ad6d5896c8b3dd97a403c40a658c1d2d1c745c6353015165c7caed5"} Oct 30 18:01:18.623767 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.623732 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-st6k6" event={"ID":"54bdc818-2b49-4f72-a168-6958864242fd","Type":"ContainerStarted","Data":"e5b4da9304f38ff8f1ec1ce493111c27348dcec5dd14255759e20fb7e8eab323"} Oct 30 18:01:18.623883 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.623778 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-st6k6" event={"ID":"54bdc818-2b49-4f72-a168-6958864242fd","Type":"ContainerStarted","Data":"64c8a192acada91c83a8aba017f0d3eb6ea73f0e74ff83b5c2793c2065074b9a"} Oct 30 18:01:18.623883 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.623864 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:18.670492 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.670432 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-b9kd2" podStartSLOduration=1.643080002 podStartE2EDuration="3.670415652s" totalImagesPullingTime="2.02733565s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:01:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:18.670116402 +0000 UTC m=+49.928975168" watchObservedRunningTime="2026-10-30 18:01:18.670415652 +0000 UTC m=+49.929274417" Oct 30 18:01:18.690006 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.689963 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.690006 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.690005 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/17beb356-34fc-4c4a-97a4-72f00c649e7b-root\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.690226 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.690031 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17beb356-34fc-4c4a-97a4-72f00c649e7b-metrics-client-ca\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.690226 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.690058 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-wtmp\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.690226 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.690096 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-textfile\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.690226 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.690122 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.690226 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.690182 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-tls\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.690226 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.690207 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grpvw\" (UniqueName: \"kubernetes.io/projected/17beb356-34fc-4c4a-97a4-72f00c649e7b-kube-api-access-grpvw\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.690495 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.690231 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/17beb356-34fc-4c4a-97a4-72f00c649e7b-root\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.690495 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.690353 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17beb356-34fc-4c4a-97a4-72f00c649e7b-sys\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.690495 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.690429 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17beb356-34fc-4c4a-97a4-72f00c649e7b-sys\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.690495 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.690483 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-wtmp\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.690717 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.690500 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-textfile\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.690717 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.690701 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17beb356-34fc-4c4a-97a4-72f00c649e7b-metrics-client-ca\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.691339 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.691312 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.693112 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.693087 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.693449 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.693430 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-tls\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.712266 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.712244 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grpvw\" (UniqueName: \"kubernetes.io/projected/17beb356-34fc-4c4a-97a4-72f00c649e7b-kube-api-access-grpvw\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.714135 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.714095 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-st6k6" podStartSLOduration=1.929763764 podStartE2EDuration="3.714082719s" totalImagesPullingTime="1.784318955s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:18.712849784 +0000 UTC m=+49.971708550" watchObservedRunningTime="2026-10-30 18:01:18.714082719 +0000 UTC m=+49.972941484" Oct 30 18:01:18.801032 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:18.800998 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:18.812572 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:01:18.812542 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17beb356_34fc_4c4a_97a4_72f00c649e7b.slice/crio-8cdd06b5b549f9a7b3257e62bbbcf1aa9df0ab85b1dc5d5c9277216b2fb491a3 WatchSource:0}: Error finding container 8cdd06b5b549f9a7b3257e62bbbcf1aa9df0ab85b1dc5d5c9277216b2fb491a3: Status 404 returned error can't find the container with id 8cdd06b5b549f9a7b3257e62bbbcf1aa9df0ab85b1dc5d5c9277216b2fb491a3 Oct 30 18:01:19.628695 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:19.628659 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-kqvrz" event={"ID":"7bd897e6-be5e-4d00-8faf-b2544c55a3f6","Type":"ContainerStarted","Data":"0b0bf7ecf80431bba46bd7a0c956c4ce780a507de3efa276cf240d749276208e"} Oct 30 18:01:19.629936 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:19.629906 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5zplp" event={"ID":"17beb356-34fc-4c4a-97a4-72f00c649e7b","Type":"ContainerStarted","Data":"8cdd06b5b549f9a7b3257e62bbbcf1aa9df0ab85b1dc5d5c9277216b2fb491a3"} Oct 30 18:01:19.670385 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:19.670322 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-kqvrz" podStartSLOduration=1.9456920100000001 podStartE2EDuration="4.670303167s" totalImagesPullingTime="2.724611157s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:19.669027173 +0000 UTC m=+50.927885940" watchObservedRunningTime="2026-10-30 18:01:19.670303167 +0000 UTC m=+50.929161933" Oct 30 18:01:19.904351 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:19.904322 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:01:19.906551 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:19.906531 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:19.930389 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:19.930281 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 18:01:19.930389 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:19.930357 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 18:01:19.930389 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:19.930366 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 18:01:19.930389 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:19.930366 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 18:01:19.930648 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:19.930492 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fqgf4\"" Oct 30 18:01:19.930648 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:19.930496 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 18:01:19.930648 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:19.930540 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 18:01:19.930648 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:19.930551 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 18:01:19.930648 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:19.930550 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 18:01:19.951319 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:19.951288 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 18:01:19.952347 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:19.952322 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:01:20.003620 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.003595 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.003731 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.003652 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.003779 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.003746 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.003823 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.003774 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75w7g\" (UniqueName: \"kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-kube-api-access-75w7g\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.003823 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.003794 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.003823 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.003812 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.003971 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.003954 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-config-volume\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.004001 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.003981 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-web-config\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.004028 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.004007 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-config-out\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.004068 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.004026 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.004104 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.004075 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.004158 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.004147 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.004188 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.004168 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.104912 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.104870 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-config-volume\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.104912 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.104905 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-web-config\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.105108 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.104928 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-config-out\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.105108 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.104954 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.106081 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.105387 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.106081 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.105481 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.106081 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.105565 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.106081 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.105619 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.106081 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.105704 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.106081 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.105745 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.106711 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.106685 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.107403 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.106972 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75w7g\" (UniqueName: \"kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-kube-api-access-75w7g\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.107403 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.107030 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.107403 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.107115 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.107403 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.107125 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.107403 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.107236 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.108595 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.108368 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-config-out\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.108784 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.108759 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-config-volume\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.109092 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.109056 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-web-config\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.109905 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.109879 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.112743 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.112471 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.112743 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.112532 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.113468 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.113439 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.113585 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.113563 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.113698 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.113677 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.126621 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.126595 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75w7g\" (UniqueName: \"kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-kube-api-access-75w7g\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.223440 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.223416 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:20.537364 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.537338 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:01:20.538295 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:01:20.538264 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb82def8_d679_4f36_8d87_b9b23865e3de.slice/crio-f1fe730a2abe2619857d71bec2084370e75929038ce5a74a4fa4b6d7113aab97 WatchSource:0}: Error finding container f1fe730a2abe2619857d71bec2084370e75929038ce5a74a4fa4b6d7113aab97: Status 404 returned error can't find the container with id f1fe730a2abe2619857d71bec2084370e75929038ce5a74a4fa4b6d7113aab97 Oct 30 18:01:20.633099 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.633061 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb82def8-d679-4f36-8d87-b9b23865e3de","Type":"ContainerStarted","Data":"f1fe730a2abe2619857d71bec2084370e75929038ce5a74a4fa4b6d7113aab97"} Oct 30 18:01:20.634570 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.634545 2577 generic.go:356] "Generic (PLEG): container finished" podID="17beb356-34fc-4c4a-97a4-72f00c649e7b" containerID="a6cc9561f4d7d29ff1bcf347db6d5dece554bd6d23f9c6e825bf1e10d9075ae3" exitCode=0 Oct 30 18:01:20.634676 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:20.634630 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5zplp" event={"ID":"17beb356-34fc-4c4a-97a4-72f00c649e7b","Type":"ContainerDied","Data":"a6cc9561f4d7d29ff1bcf347db6d5dece554bd6d23f9c6e825bf1e10d9075ae3"} Oct 30 18:01:21.641111 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:21.641078 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5zplp" event={"ID":"17beb356-34fc-4c4a-97a4-72f00c649e7b","Type":"ContainerStarted","Data":"f9b284119a77cf00e61e687cada6ba141aebe8994d0ec9c75b25bfd3e3d275ce"} Oct 30 18:01:21.641439 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:21.641154 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-5zplp" event={"ID":"17beb356-34fc-4c4a-97a4-72f00c649e7b","Type":"ContainerStarted","Data":"68672a9e968ddcfbd5789eda075209a254f2495772400f150200d167e7cc8e0b"} Oct 30 18:01:21.694306 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:21.694234 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-5zplp" podStartSLOduration=2.5936955040000003 podStartE2EDuration="3.694219382s" totalImagesPullingTime="1.021604036s" totalInitContainerRuntime="78.919842ms" isStatefulPod=true podCreationTimestamp="2026-10-30 18:01:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:21.693419452 +0000 UTC m=+52.952278219" watchObservedRunningTime="2026-10-30 18:01:21.694219382 +0000 UTC m=+52.953078148" Oct 30 18:01:22.646966 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:22.646927 2577 generic.go:356] "Generic (PLEG): container finished" podID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerID="eae7e933f2eb2fa8270d189d2f72af5847ed46a7ac5e26e71695065c7a25e4fd" exitCode=0 Oct 30 18:01:22.647407 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:22.647018 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb82def8-d679-4f36-8d87-b9b23865e3de","Type":"ContainerDied","Data":"eae7e933f2eb2fa8270d189d2f72af5847ed46a7ac5e26e71695065c7a25e4fd"} Oct 30 18:01:23.033672 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.033637 2577 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-866f94cc5d-rv8z2"] Oct 30 18:01:23.035709 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.035694 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.055784 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.055746 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 18:01:23.055784 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.055752 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 18:01:23.057069 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.057045 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-866f94cc5d-rv8z2"] Oct 30 18:01:23.129590 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.129557 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 18:01:23.129771 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.129604 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 18:01:23.129771 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.129754 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4js9k\"" Oct 30 18:01:23.129861 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.129794 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c0frpd4sn3mil\"" Oct 30 18:01:23.136190 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.136164 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.136329 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.136205 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-client-ca-bundle\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.136329 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.136231 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx4m6\" (UniqueName: \"kubernetes.io/projected/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-kube-api-access-qx4m6\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.136486 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.136357 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-secret-metrics-server-tls\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.136486 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.136397 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-metrics-server-audit-profiles\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.136486 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.136464 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-secret-metrics-server-client-certs\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.136645 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.136503 2577 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-audit-log\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.237754 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.237726 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-audit-log\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.237918 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.237767 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.237918 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.237793 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-client-ca-bundle\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.237918 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.237816 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx4m6\" (UniqueName: \"kubernetes.io/projected/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-kube-api-access-qx4m6\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.237918 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.237862 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-secret-metrics-server-tls\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.237918 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.237902 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-metrics-server-audit-profiles\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.238121 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.237946 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-secret-metrics-server-client-certs\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.238213 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.238191 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-audit-log\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.238618 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.238588 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.239138 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.239083 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-metrics-server-audit-profiles\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.241076 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.241049 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-secret-metrics-server-client-certs\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.241185 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.241161 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-client-ca-bundle\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.241254 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.241236 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-secret-metrics-server-tls\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.273368 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.273334 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx4m6\" (UniqueName: \"kubernetes.io/projected/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-kube-api-access-qx4m6\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.346490 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.346382 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:23.655819 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:23.655793 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-866f94cc5d-rv8z2"] Oct 30 18:01:23.657734 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:01:23.657705 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabadb041_5d27_42f8_9cb9_35b78ec5f3bf.slice/crio-d0b2bad00761754a9924e522685d7e4559bac531c39d895e3d1caa2b9e78aa2c WatchSource:0}: Error finding container d0b2bad00761754a9924e522685d7e4559bac531c39d895e3d1caa2b9e78aa2c: Status 404 returned error can't find the container with id d0b2bad00761754a9924e522685d7e4559bac531c39d895e3d1caa2b9e78aa2c Oct 30 18:01:24.659298 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:24.659251 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" event={"ID":"abadb041-5d27-42f8-9cb9-35b78ec5f3bf","Type":"ContainerStarted","Data":"d0b2bad00761754a9924e522685d7e4559bac531c39d895e3d1caa2b9e78aa2c"} Oct 30 18:01:24.662540 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:24.662497 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb82def8-d679-4f36-8d87-b9b23865e3de","Type":"ContainerStarted","Data":"c48d172211a010718301f9fdc8458788649f3859e0b9c223286fec5a4da6f9f4"} Oct 30 18:01:24.662662 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:24.662570 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb82def8-d679-4f36-8d87-b9b23865e3de","Type":"ContainerStarted","Data":"e417d8c7b420a656699110cdd3e941d54ee770055feddf0843d95aa95671a3f7"} Oct 30 18:01:24.662662 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:24.662597 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb82def8-d679-4f36-8d87-b9b23865e3de","Type":"ContainerStarted","Data":"402e2ba9f11434e02f7191db2e717ad08d830b74f65ecf12e8f5921302c957b6"} Oct 30 18:01:24.662662 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:24.662621 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb82def8-d679-4f36-8d87-b9b23865e3de","Type":"ContainerStarted","Data":"ff8ee0a5a16476997d930a3db3141259151ea1391a2762dd3b8f51fcb3845d46"} Oct 30 18:01:24.662662 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:24.662644 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb82def8-d679-4f36-8d87-b9b23865e3de","Type":"ContainerStarted","Data":"7c013b745e3c4229cd540ac7bb7c48da79c793355bea6343f379e52920c80797"} Oct 30 18:01:25.666971 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:25.666935 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" event={"ID":"abadb041-5d27-42f8-9cb9-35b78ec5f3bf","Type":"ContainerStarted","Data":"971cc1513cb2c3d4180e24b75c0167faf928e299fdb847c5d510b3f687ed5be4"} Oct 30 18:01:25.670276 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:25.670254 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb82def8-d679-4f36-8d87-b9b23865e3de","Type":"ContainerStarted","Data":"2e69cc3749958466d3237c968c19834fbbdd29bd8f51a3283d40ad261254985c"} Oct 30 18:01:25.713522 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:25.713456 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" podStartSLOduration=1.2955131469999999 podStartE2EDuration="2.713443363s" totalImagesPullingTime="1.417930216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:25.712801099 +0000 UTC m=+56.971659865" watchObservedRunningTime="2026-10-30 18:01:25.713443363 +0000 UTC m=+56.972302129" Oct 30 18:01:25.780128 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:25.780082 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.620860265 podStartE2EDuration="6.780065005s" totalImagesPullingTime="3.144910641s" totalInitContainerRuntime="14.294099ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:01:19 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:25.76603498 +0000 UTC m=+57.024893746" watchObservedRunningTime="2026-10-30 18:01:25.780065005 +0000 UTC m=+57.038923770" Oct 30 18:01:28.631641 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:28.631606 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:28.772888 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:28.772864 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:35.135761 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.135721 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs\") pod \"network-metrics-daemon-g8h9t\" (UID: \"914c8c09-aa3d-4543-9d91-fa443a09b1ad\") " pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:01:35.136180 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.135776 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xlp\" (UniqueName: \"kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp\") pod \"network-check-target-94r4x\" (UID: \"a3c3fe6d-7fab-4cea-a04e-693b985ceb49\") " pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:01:35.136180 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.135804 2577 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:35.151080 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.151056 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:01:35.164645 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.164623 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:01:35.164714 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.164690 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:01:35.169103 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.169081 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs\") pod \"network-metrics-daemon-g8h9t\" (UID: \"914c8c09-aa3d-4543-9d91-fa443a09b1ad\") " pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:01:35.169198 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.169080 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:35.171012 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.170992 2577 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:01:35.178333 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.178309 2577 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75xlp\" (UniqueName: \"kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp\") pod \"network-check-target-94r4x\" (UID: \"a3c3fe6d-7fab-4cea-a04e-693b985ceb49\") " pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:01:35.448810 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.448737 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khr2s\"" Oct 30 18:01:35.453901 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.453886 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:35.454793 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.454775 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qrbw2\"" Oct 30 18:01:35.460522 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.460486 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:01:35.461090 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.461071 2577 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdst5\"" Oct 30 18:01:35.466561 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.466539 2577 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:01:35.748075 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.748054 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-brgpf"] Oct 30 18:01:35.749759 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:01:35.749726 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb0df4b9d_89ce_4bb4_b759_68f1b499b52a.slice/crio-f086c1b6538ec4dc3d3e05f8caba02cc55a8c3ae8aa5cbaf84b06c90dce9f03c WatchSource:0}: Error finding container f086c1b6538ec4dc3d3e05f8caba02cc55a8c3ae8aa5cbaf84b06c90dce9f03c: Status 404 returned error can't find the container with id f086c1b6538ec4dc3d3e05f8caba02cc55a8c3ae8aa5cbaf84b06c90dce9f03c Oct 30 18:01:35.761420 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.761399 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-94r4x"] Oct 30 18:01:35.762263 ip-10-0-141-106 kubenswrapper[2577]: W1030 18:01:35.762242 2577 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3c3fe6d_7fab_4cea_a04e_693b985ceb49.slice/crio-3a6adbbde29a543a5e231d81cc1511870a1de772661753a445acbc5971f61c31 WatchSource:0}: Error finding container 3a6adbbde29a543a5e231d81cc1511870a1de772661753a445acbc5971f61c31: Status 404 returned error can't find the container with id 3a6adbbde29a543a5e231d81cc1511870a1de772661753a445acbc5971f61c31 Oct 30 18:01:35.771601 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:35.771572 2577 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-g8h9t"] Oct 30 18:01:36.705211 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:36.705171 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g8h9t" event={"ID":"914c8c09-aa3d-4543-9d91-fa443a09b1ad","Type":"ContainerStarted","Data":"096c090ce580ec649397f1ed7348b86be8c7beb2f448f8db2a38365e1f676d04"} Oct 30 18:01:36.706670 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:36.706598 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-94r4x" event={"ID":"a3c3fe6d-7fab-4cea-a04e-693b985ceb49","Type":"ContainerStarted","Data":"3a6adbbde29a543a5e231d81cc1511870a1de772661753a445acbc5971f61c31"} Oct 30 18:01:36.707926 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:36.707897 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-brgpf" event={"ID":"b0df4b9d-89ce-4bb4-b759-68f1b499b52a","Type":"ContainerStarted","Data":"f086c1b6538ec4dc3d3e05f8caba02cc55a8c3ae8aa5cbaf84b06c90dce9f03c"} Oct 30 18:01:37.714755 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:37.714710 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g8h9t" event={"ID":"914c8c09-aa3d-4543-9d91-fa443a09b1ad","Type":"ContainerStarted","Data":"53a65a65b335b6491f7aa5e174f1bdd85af2c79d2c28fcfd8d1a113cb20cfbf9"} Oct 30 18:01:37.715245 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:37.714776 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-g8h9t" event={"ID":"914c8c09-aa3d-4543-9d91-fa443a09b1ad","Type":"ContainerStarted","Data":"841f3dfaa82122c60e7d80929362bd40d335249ecba00e222be0f1ee1d9d548b"} Oct 30 18:01:37.761092 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:37.761034 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-g8h9t" podStartSLOduration=67.665314618 podStartE2EDuration="1m8.761015778s" totalImagesPullingTime="1.09570116s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:37.760092056 +0000 UTC m=+69.018950823" watchObservedRunningTime="2026-10-30 18:01:37.761015778 +0000 UTC m=+69.019874547" Oct 30 18:01:39.722741 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:39.722699 2577 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-94r4x" event={"ID":"a3c3fe6d-7fab-4cea-a04e-693b985ceb49","Type":"ContainerStarted","Data":"5c14b6176b7f2156483ac129ed2932b9f8dfc34b4d89b335704ddce7178f643e"} Oct 30 18:01:39.723216 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:39.722870 2577 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:01:39.764173 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:39.764118 2577 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-94r4x" podStartSLOduration=67.603850061 podStartE2EDuration="1m10.764106034s" totalImagesPullingTime="3.160255973s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:00:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:01:39.763848021 +0000 UTC m=+71.022706811" watchObservedRunningTime="2026-10-30 18:01:39.764106034 +0000 UTC m=+71.022964799" Oct 30 18:01:40.610401 ip-10-0-141-106 kubenswrapper[2577]: I1030 18:01:40.610270 2577 dynamic_cafile_content.go:175] "Shutting down controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:01:40.610438 ip-10-0-141-106 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 18:01:40.619858 ip-10-0-141-106 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 18:01:40.620076 ip-10-0-141-106 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 18:01:40.620140 ip-10-0-141-106 systemd[1]: kubelet.service: Consumed 3.713s CPU time. Oct 30 18:01:40.630292 ip-10-0-141-106 systemd[1]: Starting Kubernetes Kubelet... Oct 30 18:01:40.713634 ip-10-0-141-106 kubenswrapper[5232]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:01:40.713634 ip-10-0-141-106 kubenswrapper[5232]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 18:01:40.713634 ip-10-0-141-106 kubenswrapper[5232]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:01:40.713634 ip-10-0-141-106 kubenswrapper[5232]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 18:01:40.715643 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715566 5232 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:01:40.715643 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715643 5232 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:01:40.715643 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715647 5232 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715650 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715654 5232 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715657 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715660 5232 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715668 5232 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715671 5232 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715674 5232 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715676 5232 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715679 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715682 5232 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715684 5232 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715687 5232 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715689 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715692 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715694 5232 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715697 5232 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715699 5232 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715702 5232 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715705 5232 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:01:40.715739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715707 5232 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715710 5232 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715712 5232 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715716 5232 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715719 5232 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715721 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715724 5232 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715727 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715729 5232 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715732 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715734 5232 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715737 5232 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715740 5232 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715743 5232 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715746 5232 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715748 5232 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715754 5232 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715758 5232 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715761 5232 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:01:40.716220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715765 5232 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715768 5232 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715771 5232 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715774 5232 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715776 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715778 5232 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715781 5232 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715783 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715786 5232 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715789 5232 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715791 5232 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715794 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715796 5232 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715798 5232 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715801 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715803 5232 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715806 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715809 5232 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715812 5232 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715814 5232 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:01:40.716697 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715817 5232 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715819 5232 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715822 5232 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715824 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715827 5232 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715830 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715832 5232 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715835 5232 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715837 5232 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715839 5232 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715842 5232 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715845 5232 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715848 5232 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715850 5232 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715853 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715855 5232 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715858 5232 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715860 5232 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715863 5232 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715865 5232 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:01:40.717276 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715868 5232 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715870 5232 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715873 5232 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715876 5232 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715880 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715883 5232 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715886 5232 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715888 5232 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715892 5232 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715894 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715897 5232 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715899 5232 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715902 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715904 5232 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715907 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715910 5232 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715912 5232 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715916 5232 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715918 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:01:40.717796 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715921 5232 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715923 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715925 5232 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715928 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715931 5232 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.715934 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716702 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716710 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716714 5232 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716717 5232 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716720 5232 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716723 5232 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716725 5232 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716728 5232 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716730 5232 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716733 5232 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716735 5232 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716738 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716740 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:01:40.718263 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716743 5232 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716746 5232 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716749 5232 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716752 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716754 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716757 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716759 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716763 5232 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716766 5232 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716768 5232 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716770 5232 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716775 5232 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716779 5232 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716782 5232 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716784 5232 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716787 5232 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716790 5232 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716792 5232 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716795 5232 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:01:40.718735 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716798 5232 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716800 5232 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716803 5232 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716806 5232 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716808 5232 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716811 5232 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716813 5232 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716816 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716819 5232 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716821 5232 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716824 5232 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716826 5232 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716829 5232 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716832 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716834 5232 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716837 5232 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716840 5232 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716842 5232 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716845 5232 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716848 5232 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:01:40.719200 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716850 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716853 5232 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716855 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716858 5232 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716860 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716863 5232 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716866 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716869 5232 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716871 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716874 5232 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716876 5232 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716879 5232 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716881 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716884 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716886 5232 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716889 5232 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716891 5232 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716894 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716896 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716898 5232 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:01:40.719726 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716901 5232 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716904 5232 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716906 5232 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716908 5232 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716911 5232 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716914 5232 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716916 5232 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716918 5232 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716921 5232 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716924 5232 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716927 5232 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716930 5232 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716932 5232 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716934 5232 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716937 5232 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716939 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716941 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716944 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716948 5232 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:01:40.720268 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716952 5232 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716955 5232 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716959 5232 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716961 5232 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716964 5232 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716966 5232 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716969 5232 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716972 5232 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716976 5232 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716978 5232 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716981 5232 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716984 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716986 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716989 5232 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.716991 5232 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717477 5232 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717483 5232 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717487 5232 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717491 5232 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717493 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:01:40.720750 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717496 5232 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717499 5232 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717502 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717520 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717523 5232 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717525 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717528 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717531 5232 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717534 5232 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717536 5232 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717540 5232 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717542 5232 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717551 5232 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717554 5232 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717557 5232 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717559 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717562 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717564 5232 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717567 5232 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:01:40.721247 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717569 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717571 5232 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717574 5232 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717576 5232 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717579 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717583 5232 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717587 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717590 5232 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717593 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717596 5232 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717599 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717601 5232 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717604 5232 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717606 5232 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717608 5232 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717612 5232 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717614 5232 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717616 5232 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717619 5232 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:01:40.721934 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717622 5232 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717624 5232 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717627 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717629 5232 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717632 5232 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717635 5232 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717637 5232 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717640 5232 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717642 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717645 5232 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717647 5232 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717650 5232 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717652 5232 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717655 5232 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717657 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717659 5232 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717662 5232 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717664 5232 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717668 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717670 5232 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:01:40.722723 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717673 5232 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717675 5232 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717678 5232 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717680 5232 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717682 5232 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717685 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717687 5232 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717690 5232 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717694 5232 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717698 5232 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717700 5232 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717704 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717706 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717709 5232 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717711 5232 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717714 5232 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717716 5232 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717719 5232 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717721 5232 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:01:40.723326 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717723 5232 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717728 5232 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717731 5232 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717733 5232 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717736 5232 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717738 5232 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717741 5232 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717743 5232 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717745 5232 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717748 5232 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717754 5232 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717757 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717759 5232 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717762 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717764 5232 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717767 5232 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717770 5232 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717772 5232 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717775 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717777 5232 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:01:40.723815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717780 5232 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717782 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717785 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.717788 5232 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717876 5232 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717885 5232 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717893 5232 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717899 5232 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717905 5232 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717910 5232 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717916 5232 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717921 5232 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717924 5232 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717927 5232 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717931 5232 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717935 5232 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717938 5232 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717941 5232 flags.go:64] FLAG: --cgroup-root="" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717944 5232 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717947 5232 flags.go:64] FLAG: --client-ca-file="" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717949 5232 flags.go:64] FLAG: --cloud-provider="external" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717952 5232 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717958 5232 flags.go:64] FLAG: --cluster-domain="" Oct 30 18:01:40.724296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717961 5232 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717964 5232 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717968 5232 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717971 5232 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717975 5232 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717978 5232 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717981 5232 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717984 5232 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717987 5232 flags.go:64] FLAG: --contention-profiling="false" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717990 5232 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717993 5232 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717996 5232 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.717999 5232 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718003 5232 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718007 5232 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718010 5232 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718013 5232 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718016 5232 flags.go:64] FLAG: --enable-server="true" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718019 5232 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718023 5232 flags.go:64] FLAG: --event-burst="100" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718026 5232 flags.go:64] FLAG: --event-qps="50" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718029 5232 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718032 5232 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718035 5232 flags.go:64] FLAG: --eviction-hard="" Oct 30 18:01:40.724876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718039 5232 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718042 5232 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718045 5232 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718048 5232 flags.go:64] FLAG: --eviction-soft="" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718051 5232 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718054 5232 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718057 5232 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718060 5232 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718063 5232 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718066 5232 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718069 5232 flags.go:64] FLAG: --feature-gates="" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718073 5232 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718076 5232 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718079 5232 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718082 5232 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718085 5232 flags.go:64] FLAG: --healthz-port="10248" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718088 5232 flags.go:64] FLAG: --help="false" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718091 5232 flags.go:64] FLAG: --hostname-override="ip-10-0-141-106.ec2.internal" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718094 5232 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718097 5232 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718100 5232 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718103 5232 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718110 5232 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718113 5232 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 18:01:40.725477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718115 5232 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718118 5232 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718121 5232 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718124 5232 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718127 5232 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718130 5232 flags.go:64] FLAG: --kube-reserved="" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718133 5232 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718136 5232 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718139 5232 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718142 5232 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718145 5232 flags.go:64] FLAG: --lock-file="" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718148 5232 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718150 5232 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718154 5232 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718159 5232 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718163 5232 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718167 5232 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718169 5232 flags.go:64] FLAG: --logging-format="text" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718172 5232 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718176 5232 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718178 5232 flags.go:64] FLAG: --manifest-url="" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718181 5232 flags.go:64] FLAG: --manifest-url-header="" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718186 5232 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718189 5232 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718192 5232 flags.go:64] FLAG: --max-pods="110" Oct 30 18:01:40.726067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718195 5232 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718198 5232 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718201 5232 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718204 5232 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718207 5232 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718211 5232 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718214 5232 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718221 5232 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718224 5232 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718227 5232 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718230 5232 flags.go:64] FLAG: --pod-cidr="" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718233 5232 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718236 5232 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718239 5232 flags.go:64] FLAG: --pods-per-core="0" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718242 5232 flags.go:64] FLAG: --port="10250" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718245 5232 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718248 5232 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-09e88f591e009548a" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718251 5232 flags.go:64] FLAG: --qos-reserved="" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718254 5232 flags.go:64] FLAG: --read-only-port="10255" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718257 5232 flags.go:64] FLAG: --register-node="true" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718260 5232 flags.go:64] FLAG: --register-with-taints="" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718264 5232 flags.go:64] FLAG: --registry-burst="10" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718268 5232 flags.go:64] FLAG: --registry-qps="5" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718270 5232 flags.go:64] FLAG: --reserved-cpus="" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718273 5232 flags.go:64] FLAG: --reserved-memory="" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718277 5232 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 18:01:40.726778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718280 5232 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718283 5232 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718286 5232 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718289 5232 flags.go:64] FLAG: --runonce="false" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718292 5232 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718295 5232 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718298 5232 flags.go:64] FLAG: --seccomp-default="false" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718301 5232 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718304 5232 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718307 5232 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718310 5232 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718314 5232 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718317 5232 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718320 5232 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718322 5232 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718325 5232 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718328 5232 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718331 5232 flags.go:64] FLAG: --system-cgroups="" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718334 5232 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718339 5232 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718342 5232 flags.go:64] FLAG: --tls-cert-file="" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718345 5232 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718349 5232 flags.go:64] FLAG: --tls-min-version="" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718352 5232 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718355 5232 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 18:01:40.727428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718358 5232 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718361 5232 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718364 5232 flags.go:64] FLAG: --v="2" Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718369 5232 flags.go:64] FLAG: --version="false" Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718373 5232 flags.go:64] FLAG: --vmodule="" Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718377 5232 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718380 5232 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718520 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718524 5232 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718527 5232 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718530 5232 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718533 5232 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718536 5232 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718538 5232 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718541 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718544 5232 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718546 5232 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718549 5232 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718553 5232 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718555 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718558 5232 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:01:40.728259 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718560 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718563 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718566 5232 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718568 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718571 5232 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718574 5232 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718576 5232 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718579 5232 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718581 5232 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718584 5232 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718588 5232 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718592 5232 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718595 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718598 5232 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718601 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718604 5232 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718607 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718609 5232 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718612 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:01:40.729154 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718614 5232 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718617 5232 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718619 5232 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718622 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718625 5232 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718627 5232 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718630 5232 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718632 5232 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718635 5232 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718637 5232 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718641 5232 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718644 5232 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718646 5232 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718649 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718651 5232 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718654 5232 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718657 5232 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718659 5232 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718662 5232 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718664 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:01:40.729793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718667 5232 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718669 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718672 5232 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718674 5232 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718677 5232 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718679 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718682 5232 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718686 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718689 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718691 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718695 5232 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718698 5232 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718701 5232 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718704 5232 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718707 5232 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718710 5232 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718712 5232 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718715 5232 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718717 5232 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:01:40.730335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718719 5232 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718722 5232 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718725 5232 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718728 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718730 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718733 5232 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718735 5232 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718738 5232 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718741 5232 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718744 5232 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718746 5232 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718749 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718751 5232 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718754 5232 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718756 5232 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718759 5232 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718761 5232 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718764 5232 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718766 5232 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718769 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:01:40.730850 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718773 5232 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:01:40.731335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718776 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:01:40.731335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718778 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:01:40.731335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718781 5232 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:01:40.731335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718783 5232 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:01:40.731335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718786 5232 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:01:40.731335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718788 5232 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:01:40.731335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718791 5232 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:01:40.731335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718793 5232 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:01:40.731335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718796 5232 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:01:40.731335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718798 5232 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:01:40.731335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718801 5232 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:01:40.731335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718804 5232 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:01:40.731335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718806 5232 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:01:40.731335 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.718831 5232 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:01:40.731335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718868 5232 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:01:40.731335 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718872 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718875 5232 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718877 5232 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718880 5232 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718883 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718886 5232 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718888 5232 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718891 5232 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718893 5232 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718896 5232 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718898 5232 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718901 5232 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718904 5232 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718906 5232 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718909 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718911 5232 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718914 5232 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718918 5232 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718921 5232 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718923 5232 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:01:40.731778 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718926 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718928 5232 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718931 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718933 5232 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718936 5232 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718938 5232 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718941 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718943 5232 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718945 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718948 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718951 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718955 5232 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718959 5232 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718962 5232 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718964 5232 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718967 5232 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718969 5232 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718971 5232 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718974 5232 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:01:40.732450 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718976 5232 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718979 5232 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718981 5232 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718984 5232 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718986 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718988 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718991 5232 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718993 5232 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718996 5232 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.718998 5232 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719002 5232 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719005 5232 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719007 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719010 5232 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719012 5232 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719015 5232 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719017 5232 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719020 5232 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719022 5232 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719025 5232 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:01:40.732983 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719027 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719029 5232 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719032 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719034 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719038 5232 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719041 5232 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719043 5232 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719046 5232 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719048 5232 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719050 5232 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719053 5232 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719055 5232 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719058 5232 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719060 5232 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719063 5232 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719066 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719069 5232 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719071 5232 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719074 5232 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719076 5232 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:01:40.733452 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719078 5232 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719081 5232 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719085 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719088 5232 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719090 5232 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719093 5232 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719095 5232 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719098 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719100 5232 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719103 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719105 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719108 5232 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719110 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719113 5232 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719115 5232 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719118 5232 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719122 5232 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719124 5232 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719127 5232 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:01:40.734045 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719130 5232 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719133 5232 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719136 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719139 5232 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719142 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719144 5232 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.719146 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.724082 5232 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.724099 5232 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.724122 5232 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.724127 5232 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724185 5232 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724191 5232 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724195 5232 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724198 5232 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724201 5232 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724204 5232 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724206 5232 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724209 5232 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:01:40.734550 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724212 5232 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724214 5232 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724217 5232 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724220 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724222 5232 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724225 5232 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724230 5232 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724233 5232 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724236 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724239 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724241 5232 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724244 5232 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724247 5232 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724250 5232 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724252 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724255 5232 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724257 5232 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724260 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724263 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724265 5232 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:01:40.735043 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724268 5232 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724271 5232 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724274 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724276 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724279 5232 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724281 5232 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724283 5232 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724286 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724288 5232 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724291 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724293 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724296 5232 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724298 5232 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724301 5232 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724304 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724306 5232 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724309 5232 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724313 5232 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724322 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:01:40.735543 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724325 5232 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724328 5232 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724331 5232 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724334 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724337 5232 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724340 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724342 5232 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724345 5232 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724347 5232 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724350 5232 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724352 5232 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724355 5232 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724357 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724360 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724362 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724365 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724367 5232 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724370 5232 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724373 5232 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:01:40.736003 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724375 5232 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724378 5232 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724380 5232 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724383 5232 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724385 5232 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724387 5232 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724390 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724394 5232 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724398 5232 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724400 5232 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724403 5232 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724406 5232 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724409 5232 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724412 5232 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724415 5232 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724418 5232 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724421 5232 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724424 5232 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724427 5232 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:01:40.736976 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724429 5232 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724432 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724434 5232 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724437 5232 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724440 5232 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724442 5232 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724445 5232 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724447 5232 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724450 5232 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724452 5232 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724455 5232 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724457 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724460 5232 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724463 5232 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724465 5232 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724468 5232 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724470 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724473 5232 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724475 5232 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724478 5232 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:01:40.737702 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724480 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:01:40.738199 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.724520 5232 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:01:40.738199 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724659 5232 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:01:40.738199 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724665 5232 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:01:40.738199 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724668 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:01:40.738199 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724670 5232 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:01:40.738199 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724673 5232 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:01:40.738199 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724676 5232 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:01:40.738199 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724679 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:01:40.738199 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724682 5232 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:01:40.738199 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724684 5232 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:01:40.738199 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724687 5232 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:01:40.738199 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724689 5232 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:01:40.738199 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724692 5232 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:01:40.738199 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724695 5232 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:01:40.738199 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724700 5232 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:01:40.738199 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724702 5232 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724705 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724708 5232 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724710 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724713 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724715 5232 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724717 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724720 5232 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724722 5232 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724725 5232 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724727 5232 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724730 5232 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724732 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724735 5232 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724737 5232 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724739 5232 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724742 5232 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724744 5232 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724747 5232 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:01:40.738617 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724749 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724752 5232 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724754 5232 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724757 5232 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724759 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724762 5232 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724765 5232 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724767 5232 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724769 5232 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724772 5232 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724774 5232 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724777 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724779 5232 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724782 5232 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724784 5232 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724787 5232 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724789 5232 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724792 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724794 5232 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:01:40.739115 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724796 5232 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724799 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724801 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724804 5232 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724806 5232 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724809 5232 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724811 5232 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724813 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724816 5232 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724818 5232 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724820 5232 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724823 5232 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724826 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724828 5232 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724831 5232 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724833 5232 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724836 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724838 5232 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724841 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724844 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:01:40.739867 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724847 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724849 5232 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724851 5232 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724854 5232 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724856 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724859 5232 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724861 5232 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724864 5232 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724866 5232 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724869 5232 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724871 5232 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724873 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724876 5232 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724878 5232 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724881 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724884 5232 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724886 5232 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724889 5232 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724891 5232 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724894 5232 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:01:40.740484 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724897 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:01:40.741205 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724899 5232 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:01:40.741205 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724901 5232 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:01:40.741205 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724904 5232 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:01:40.741205 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724906 5232 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:01:40.741205 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724909 5232 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:01:40.741205 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724911 5232 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:01:40.741205 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724914 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:01:40.741205 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724917 5232 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:01:40.741205 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724920 5232 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:01:40.741205 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724923 5232 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:01:40.741205 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724926 5232 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:01:40.741205 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724929 5232 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:01:40.741205 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724931 5232 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:01:40.741205 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.724954 5232 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 18:01:40.741205 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724990 5232 feature_gate.go:386] unrecognized feature gate: Example Oct 30 18:01:40.741205 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724994 5232 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.724997 5232 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725000 5232 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725003 5232 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725006 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725009 5232 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725012 5232 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725015 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725017 5232 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725020 5232 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725023 5232 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725025 5232 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725028 5232 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725030 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725033 5232 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725035 5232 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725037 5232 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725040 5232 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725042 5232 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725045 5232 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 18:01:40.741845 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725047 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725051 5232 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725055 5232 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725058 5232 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725061 5232 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725063 5232 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725066 5232 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725068 5232 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725071 5232 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725074 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725076 5232 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725079 5232 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725081 5232 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725084 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725087 5232 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725089 5232 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725092 5232 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725094 5232 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725097 5232 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 18:01:40.742520 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725100 5232 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725104 5232 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725107 5232 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725109 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725112 5232 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725115 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725117 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725120 5232 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725124 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725127 5232 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725129 5232 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725132 5232 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725135 5232 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725137 5232 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725139 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725143 5232 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725145 5232 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725150 5232 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725153 5232 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 18:01:40.743000 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725155 5232 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725158 5232 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725160 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725163 5232 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725167 5232 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725169 5232 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725172 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725174 5232 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725177 5232 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725179 5232 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725182 5232 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725185 5232 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725187 5232 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725190 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725192 5232 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725195 5232 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725197 5232 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725199 5232 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725202 5232 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 18:01:40.743451 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725204 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725207 5232 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725211 5232 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725214 5232 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725217 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725220 5232 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725222 5232 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725225 5232 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725227 5232 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725229 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725232 5232 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725235 5232 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725238 5232 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725240 5232 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725243 5232 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725245 5232 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725248 5232 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725250 5232 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725253 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725256 5232 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 18:01:40.743956 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725259 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 18:01:40.744652 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725261 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 18:01:40.744652 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725263 5232 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 18:01:40.744652 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725266 5232 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 18:01:40.744652 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725269 5232 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 18:01:40.744652 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725272 5232 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 18:01:40.744652 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725274 5232 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 18:01:40.744652 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:01:40.725277 5232 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 18:01:40.744652 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.725333 5232 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 18:01:40.744652 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.726258 5232 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 18:01:40.744652 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.726312 5232 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 18:01:40.744652 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.726490 5232 server.go:1014] "Starting client certificate rotation" Oct 30 18:01:40.744652 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.726639 5232 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 18:01:40.744652 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.726692 5232 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 17:55:29 +0000 UTC" deadline="2028-07-02 08:28:46.953877991 +0000 UTC" Oct 30 18:01:40.744652 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.726727 5232 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14654h27m6.227152981s" Oct 30 18:01:40.744652 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.726944 5232 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:01:40.745252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.727971 5232 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 18:01:40.745252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.729754 5232 log.go:25] "Validated CRI v1 runtime API" Oct 30 18:01:40.745252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.731673 5232 log.go:25] "Validated CRI v1 image API" Oct 30 18:01:40.745252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.732407 5232 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 18:01:40.745252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.734323 5232 fs.go:135] Filesystem UUIDs: map[7B77-95E7:/dev/nvme0n1p2 b4cf3e42-5869-4504-aa36-c7470c8ef023:/dev/nvme0n1p4 bb9e4620-dff8-49c9-b0d0-97064b75932a:/dev/nvme0n1p3] Oct 30 18:01:40.745420 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.734338 5232 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/03aabceb0285bf62c690b8cbb84759a7f4ceaaca8894efab90b4da611298d336/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/03aabceb0285bf62c690b8cbb84759a7f4ceaaca8894efab90b4da611298d336/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/096c090ce580ec649397f1ed7348b86be8c7beb2f448f8db2a38365e1f676d04/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/096c090ce580ec649397f1ed7348b86be8c7beb2f448f8db2a38365e1f676d04/userdata/shm major:0 minor:333 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0c1b111e1b4322013cec7a18f50d9149c78abb670cb2e693767430c964ffc492/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0c1b111e1b4322013cec7a18f50d9149c78abb670cb2e693767430c964ffc492/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/29d1f2e2c17db36f7f56db2256ddbe2cae8e4cbc5dfce83649dae19e0c121e49/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/29d1f2e2c17db36f7f56db2256ddbe2cae8e4cbc5dfce83649dae19e0c121e49/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3a6adbbde29a543a5e231d81cc1511870a1de772661753a445acbc5971f61c31/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3a6adbbde29a543a5e231d81cc1511870a1de772661753a445acbc5971f61c31/userdata/shm major:0 minor:331 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5350ba5b6e2fc68a67e5d0f1eb3a7426e97c5a39f651ed171b95e2e062cf719e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5350ba5b6e2fc68a67e5d0f1eb3a7426e97c5a39f651ed171b95e2e062cf719e/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/8cdd06b5b549f9a7b3257e62bbbcf1aa9df0ab85b1dc5d5c9277216b2fb491a3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/8cdd06b5b549f9a7b3257e62bbbcf1aa9df0ab85b1dc5d5c9277216b2fb491a3/userdata/shm major:0 minor:247 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9c7787af32f326b1a33fc8969a4fdee0caa75dfa0a53db231bc1d4dc60e524db/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9c7787af32f326b1a33fc8969a4fdee0caa75dfa0a53db231bc1d4dc60e524db/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/9dbc196724d7a247bbf69f1d92235f3cb08b6e2a86590d7a7b9e729289a7ddfc/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/9dbc196724d7a247bbf69f1d92235f3cb08b6e2a86590d7a7b9e729289a7ddfc/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a9dc368ba69895cdc26db91f28d278559f01fa3b3fbf328629be8e9be071331f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a9dc368ba69895cdc26db91f28d278559f01fa3b3fbf328629be8e9be071331f/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b0259f4e815dae1a810bd384132879d3c437f9ee8cca15cdf54a11e49ce033e9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b0259f4e815dae1a810bd384132879d3c437f9ee8cca15cdf54a11e49ce033e9/userdata/shm major:0 minor:77 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/b1a5bdb79ebf602be71a30fb617e07ad2b711b14427053b18a752e2bfd0a19ed/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/b1a5bdb79ebf602be71a30fb617e07ad2b711b14427053b18a752e2bfd0a19ed/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/bf6f41d2257c25cec09e37516e4d3dee5a780bc4239b38af7ec4540e8b434c4e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/bf6f41d2257c25cec09e37516e4d3dee5a780bc4239b38af7ec4540e8b434c4e/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d0b2bad00761754a9924e522685d7e4559bac531c39d895e3d1caa2b9e78aa2c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d0b2bad00761754a9924e522685d7e4559bac531c39d895e3d1caa2b9e78aa2c/userdata/shm major:0 minor:285 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ed434e77d9eaa7fcb359f40da57c50494b5b63005a9e002add9d7ee20df0a13c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ed434e77d9eaa7fcb359f40da57c50494b5b63005a9e002add9d7ee20df0a13c/userdata/shm major:0 minor:51 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f086c1b6538ec4dc3d3e05f8caba02cc55a8c3ae8aa5cbaf84b06c90dce9f03c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f086c1b6538ec4dc3d3e05f8caba02cc55a8c3ae8aa5cbaf84b06c90dce9f03c/userdata/shm major:0 minor:329 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f088090189782932f4518a0624192c82cea30262310c71044c8227e22c76e8a5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f088090189782932f4518a0624192c82cea30262310c71044c8227e22c76e8a5/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1fe730a2abe2619857d71bec2084370e75929038ce5a74a4fa4b6d7113aab97/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1fe730a2abe2619857d71bec2084370e75929038ce5a74a4fa4b6d7113aab97/userdata/shm major:0 minor:267 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ffa54663ed165a1dbaec40fa9a7534778162d5b3fe9ee8e0ec42583267dbec8e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ffa54663ed165a1dbaec40fa9a7534778162d5b3fe9ee8e0ec42583267dbec8e/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/11311f90-d76a-42b5-b6a2-35c0b62bd222/volumes/kubernetes.io~projected/kube-api-access-bfmlm:{mountpoint:/var/lib/kubelet/pods/11311f90-d76a-42b5-b6a2-35c0b62bd222/volumes/kubernetes.io~projected/kube-api-access-bfmlm major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17beb356-34fc-4c4a-97a4-72f00c649e7b/volumes/kubernetes.io~projected/kube-api-access-grpvw:{mountpoint:/var/lib/kubelet/pods/17beb356-34fc-4c4a-97a4-72f00c649e7b/volumes/kubernetes.io~projected/kube-api-access-grpvw major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17beb356-34fc-4c4a-97a4-72f00c649e7b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/17beb356-34fc-4c4a-97a4-72f00c649e7b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/17beb356-34fc-4c4a-97a4-72f00c649e7b/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/17beb356-34fc-4c4a-97a4-72f00c649e7b/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31af1374-0d44-4636-bb2a-8e3836ac56b2/volumes/kubernetes.io~projected/kube-api-access-bhp2k:{mountpoint:/var/lib/kubelet/pods/31af1374-0d44-4636-bb2a-8e3836ac56b2/volumes/kubernetes.io~projected/kube-api-access-bhp2k major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/31af1374-0d44-4636-bb2a-8e3836ac56b2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/31af1374-0d44-4636-bb2a-8e3836ac56b2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/40dd1611-7ec9-411f-920d-e79df7273b90/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/40dd1611-7ec9-411f-920d-e79df7273b90/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4627766d-20a1-46cd-baf9-cff7cc120679/volumes/kubernetes.io~projected/kube-api-access-q4xwn:{mountpoint:/var/lib/kubelet/pods/4627766d-20a1-46cd-baf9-cff7cc120679/volumes/kubernetes.io~projected/kube-api-access-q4xwn major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54bdc818-2b49-4f72-a168-6958864242fd/volumes/kubernetes.io~projected/kube-api-access-8clmh:{mountpoint:/var/lib/kubelet/pods/54bdc818-2b49-4f72-a168-6958864242fd/volumes/kubernetes.io~projected/kube-api-access-8clmh major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54bdc818-2b49-4f72-a168-6958864242fd/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/54bdc818-2b49-4f72-a168-6958864242fd/volumes/kubernetes.io~secret/metrics-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc/volumes/kubernetes.io~projected/kube-api-access-pkx2s:{mountpoint:/var/lib/kubelet/pods/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc/volumes/kubernetes.io~projected/kube-api-access-pkx2s major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5e5fca56-5e6b-404d-84b2-4a876f5034cd/volumes/kubernetes.io~projected/kube-api-access-rsj7n:{mountpoint:/var/lib/kubelet/pods/5e5fca56-5e6b-404d-84b2-4a876f5034cd/volumes/kubernetes.io~projected/kube-api-access-rsj7n major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d458e1b-228c-4e8e-b66f-9688583b39db/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/6d458e1b-228c-4e8e-b66f-9688583b39db/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d458e1b-228c-4e8e-b66f-9688583b39db/volumes/kubernetes.io~projected/kube-api-access-pm2mh:{mountpoint:/var/lib/kubelet/pods/6d458e1b-228c-4e8e-b66f-9688583b39db/volumes/kubernetes.io~projected/kube-api-access-pm2mh major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6d458e1b-228c-4e8e-b66f-9688583b39db/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/6d458e1b-228c-4e8e-b66f-9688583b39db/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76146060-cfc6-4765-82a3-5c5c7601dfda/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/76146060-cfc6-4765-82a3-5c5c7601dfda/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76146060-cfc6-4765-82a3-5c5c7601dfda/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/76146060-cfc6-4765-82a3-5c5c7601dfda/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/76146060-cfc6-4765-82a3-5c5c7601dfda/volumes/kubernetes.io~projected/kube-api-access-p496j:{mountpoint:/var/lib/kubelet/pods/76146060-cfc6-4765-82a3-5c5c7601dfda/volumes/kubernetes.io~projected/kube-api-access-p496j major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bd897e6-be5e-4d00-8faf-b2544c55a3f6/volumes/kubernetes.io~projected/kube-api-access-4rjmx:{mountpoint:/var/lib/kubelet/pods/7bd897e6-be5e-4d00-8faf-b2544c55a3f6/volumes/kubernetes.io~projected/kube-api-access-4rjmx major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/7bd897e6-be5e-4d00-8faf-b2544c55a3f6/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/7bd897e6-be5e-4d00-8faf-b2544c55a3f6/volumes/kubernetes.io~secret/cert major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8754b620-b515-4ee4-9055-469a81126c9e/volumes/kubernetes.io~projected/kube-api-access-2nstb:{mountpoint:/var/lib/kubelet/pods/8754b620-b515-4ee4-9055-469a81126c9e/volumes/kubernetes.io~projected/kube-api-access-2nstb major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8d711c2d-0597-4d83-a884-83e50d98378c/volumes/kubernetes.io~projected/kube-api-access-rbr88:{mountpoint:/var/lib/kubelet/pods/8d711c2d-0597-4d83-a884-83e50d98378c/volumes/kubernetes.io~projected/kube-api-access-rbr88 major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/914c8c09-aa3d-4543-9d91-fa443a09b1ad/volumes/kubernetes.io~projected/kube-api-access-dtfcr:{mountpoint:/var/lib/kubelet/pods/914c8c09-aa3d-4543-9d91-fa443a09b1ad/volumes/kubernetes.io~projected/kube-api-access-dtfcr major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/914c8c09-aa3d-4543-9d91-fa443a09b1ad/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/914c8c09-aa3d-4543-9d91-fa443a09b1ad/volumes/kubernetes.io~secret/metrics-certs major:0 minor:322 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a3c3fe6d-7fab-4cea-a04e-693b985ceb49/volumes/kubernetes.io~projected/kube-api-access-75xlp:{mountpoint:/var/lib/kubelet/pods/a3c3fe6d-7fab-4cea-a04e-693b985ceb49/volumes/kubernetes.io~projected/kube-api-access-75xlp major:0 minor:328 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abadb041-5d27-42f8-9cb9-35b78ec5f3bf/volumes/kubernetes.io~projected/kube-api-access-qx4m6:{mountpoint:/var/lib/kubelet/pods/abadb041-5d27-42f8-9cb9-35b78ec5f3bf/volumes/kubernetes.io~projected/kube-api-access-qx4m6 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abadb041-5d27-42f8-9cb9-35b78ec5f3bf/volumes/kubernetes.io~secret/client-ca-bundle:{mountpoint:/var/lib/kubelet/pods/abadb041-5d27-42f8-9cb9-35b78ec5f3bf/volumes/kubernetes.io~secret/client-ca-bundle major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abadb041-5d27-42f8-9cb9-35b78ec5f3bf/volumes/kubernetes.io~secret/secret-metrics-server-client-certs:{mountpoint:/var/lib/kubelet/pods/abadb041-5d27-42f8-9cb9-35b78ec5f3bf/volumes/kubernetes.io~secret/secret-metrics-server-client-certs major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/abadb041-5d27-42f8-9cb9-35b78ec5f3bf/volumes/kubernetes.io~secret/secret-metrics-server-tls:{mountpoint:/var/lib/kubelet/pods/abadb041-5d27-42f8-9cb9-35b78ec5f3bf/volumes/kubernetes.io~secret/secret-metrics-server-tls major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b0df4b9d-89ce-4bb4-b759-68f1b499b52a/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/b0df4b9d-89ce-4bb4-b759-68f1b499b52a/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:327 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volume-subpaths/web-config/alertmanager/9 major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volume-subpaths/web-config/config-reloader/6 major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~empty-dir/config-out major:0 minor:253 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~projected/kube-api-access-75w7g:{mountpoint:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~projected/kube-api-access-75w7g major:0 minor:266 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~projected/tls-assets major:0 minor:260 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:262 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/config-volume major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:265 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:261 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:263 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:264 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/web-config major:0 minor:259 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/ebc82b856b39fa04fc55abb07149221ad74375e741207b75815f8e49f3f6bb0f/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/d72ec170142399a7443f55e03d3d920f7a7f66132ddccfe31df5f4dc0f0fca5b/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/e0b994ebef68ee15b158ee32740647544ab67158a2660b1bc0156376277e4054/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/bb5c84b87925de4a6779664144d4b95c478ac55d245d7ebae51013a44e2d4aec/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/2e41baf32922c1243729eaf0538fedc544f680e05212857aa8b8f7d1ccd62868/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/bffb8c4476e40661fa15a0d8417b2ae17c384ee3e5c883aa0ca2d577d903a351/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/e64686a9710572511a76ec6a54813e49af68901dc386dbcabd1937cb010d587c/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/45dc0f6bbbc208a7595efc6df91ae5db6270c72c84c3da0903916db1dc04cf26/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/aa90b6d38b8e7f45b06d49a3c4de24d49950a3a18e64e8a159e4de0e425877ff/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-134:{mountpoint:/var/lib/containers/storage/overlay/a9680923f01ea2f8d5c2bbaf2588f201c0eef7e0426bb888f562b41a926523f5/merged major:0 minor:134 fsType:overlay blockSize:0} overlay_0-142:{mountpoint:/var/lib/containers/storage/overlay/f7a35b227752ce216af2be7ab10a869e254c565f7a0f762bebae467cc4f22252/merged major:0 minor:142 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/136f62e5b05ead91f0ace2722c811e0695c57234e2348cbc081ea080410838da/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/5426abc70d4155401f8087758cf715c743880cd0c8625a74296305078c7e6329/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/1dd244dfe197d9cdbf24ad12eec67391337b2ba1943f36ac060cfa9f0b747f51/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/c82e9aaf63d47afd13ba6c78da3a3fe0ea2b29d6a2376eb95189f5a3e0c46cdc/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1ff2cd8130e704e20535fee696afbffb22bce79f2b4ba5b1c7853ec17a47e45e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/175b464dd5308b386d92cc1f9278a2d57dfb8fa1fa849a16340154c705b88a18/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/618ec765f52154b3a7dc246b130b09fc2e717f9cb85b81b3cdcefef3cfc4ea46/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/57e8346fb4eec8e6b5fe6332e6fefc15c2833e6a995965a8a5b4a7b22462023f/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/76fe1c75b483a4daf397e2b8938a60281374e83330dfd83d519839350c627b7f/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/f3645abd3aa4a848477c56d63a42a274d948750a74ae7fea05ded2b791611795/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/68ae9c4eff99c976a38ee0c94f5e198e192f3434fd43e81996aac0bf66063c83/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/a6eef1677068a70d774e56ea784c28b8457490fd8b429bf8391769548a45b585/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/3688a92e26578a74e51eb3af84baa79ef64799d7e53b78cd8ebfb920dcc05b16/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/0a100b0852d645c06821ac7ff6b7199d179d9d7a17d15640a86c0a12fe5d2b61/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/206a5f4188b275555832c34c0fe59f5751fe9cbe6d466da67980498d501abda4/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/b474bd16a0055e57b6e586a5f895fe2cc903bae0598d945e2907ad1e637fd507/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/7baa3e636e6d5e5b78eec8a26052383022303206031cefcbf13940d532b14ab7/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/b86f333ed9ecc279e89bed276f61c8a6b38cc8a7c0c4e0cebfe4283265a3754f/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-249:{mountpoint:/var/lib/containers/storage/overlay/7485c1abb8650d710845b18cc6c9d27a8760f1c7657e99c07bb6d15fb07732e7/merged major:0 minor:249 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/90176c89007e872c49fe358919b290d961e759149e4dcc8ea26ac6a6ae851a32/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-269:{mountpoint:/var/lib/containers/storage/overlay/b873d8e827ba557ef93870bd4dec134366e51aaefb9bf64cca8b05671b0753fd/merged major:0 minor:269 fsType:overlay blockSize:0} overlay_0-271:{mountpoint:/var/lib/containers/storage/overlay/827c164215f5f029d3625cfa17f914f65f66014609b54a008c98efab75bf59f4/merged major:0 minor:271 fsType:overlay blockSize:0} overlay_0-273:{mountpoint:/var/lib/containers/storage/overlay/40f02212670259d844907bfa23f4a292fcd9f69c18f84bd643fb452e6c87e390/merged major:0 minor:273 fsType:overlay blockSize:0} overlay_0-287:{mountpoint:/var/lib/containers/storage/overlay/576512b16f2e6044ad384334b29a56e1563a109dee85a44ea7816b598f3b818d/merged major:0 minor:287 fsType:overlay blockSize:0} overlay_0-289:{mountpoint:/var/lib/containers/storage/overlay/1256d81c35bc15428b1e4ba7ee922620acc47ccbbef66536b08f4d9df5fe6a25/merged major:0 minor:289 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/529ccc1901041356632c0a85bdecf48a25feb3e7f67ac988cfb924546ba2dc86/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/287194087c949a69faee7b2851c685a3f018de7d5724dfef3d50cbe3095bd0f6/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/0bbcae81aec57f33bbda326981e40d309d67ba8a82ee93062803adb67ae88ba8/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-307:{mountpoint:/var/lib/containers/storage/overlay/b7ed93449ccb5e2015055ac2438db06bdbb27ef414cdc0b376ec35ba05ddd816/merged major:0 minor:307 fsType:overlay blockSize:0} overlay_0-312:{mountpoint:/var/lib/containers/storage/overlay/ceae3bff55d55d26cc3671eb1c2f360de4b0ae925a7985b764b00e55d642d685/merged major:0 minor:312 fsType:overlay blockSize:0} overlay_0-317:{mountpoint:/var/lib/containers/storage/overlay/7b3120a8fd0a0e35c9fedace66ef289c0e35e30407e6507cd4d6e0a3ad489f6a/merged major:0 minor:317 fsType:overlay blockSize:0} overlay_0-335:{mountpoint:/var/lib/containers/storage/overlay/d99b95be34823cde1f1b292b19562e0e687660882a31490385e2c443c1cf6c3f/merged major:0 minor:335 fsType:overlay blockSize:0} overlay_0-337:{mountpoint:/var/lib/containers/storage/overlay/74bca95e5d4d54e7358fc4e05f514b9a131c583525d2e5118255a2cb2cd5bd94/merged major:0 minor:337 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/8af9df107fcb27884268afd000840602d073ff74e79b77fbfc3165a002059561/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-341:{mountpoint:/var/lib/containers/storage/overlay/d6131cfe57413902d439ae3046c23f4286d450ddb1b369c895f8688502be6c64/merged major:0 minor:341 fsType:overlay blockSize:0} overlay_0-343:{mountpoint:/var/lib/containers/storage/overlay/469cacb8cd530d5bd93c2fa6e1d856574858142c7324da209dadcde9fd17b865/merged major:0 minor:343 fsType:overlay blockSize:0} overlay_0-349:{mountpoint:/var/lib/containers/storage/overlay/96bb25b9fe75cd382bf2821314913ac97c098da7f179feb95751e017f2d3d2b8/merged major:0 minor:349 fsType:overlay blockSize:0} overlay_0-354:{mountpoint:/var/lib/containers/storage/overlay/92b0821a59fe372b6f876ad26984da9e7dc24e0171a2c83292f521a584c4a0a4/merged major:0 minor:354 fsType:overlay blockSize:0} overlay_0-49:{mountpoint:/var/lib/containers/storage/overlay/27e81e5e99dc589610897882b2302ad6fc4bb4d13ff8ed0934edaa6fc94c6561/merged major:0 minor:49 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/63aecb982c6a6c013d3dcdff7fbd6f6f451dcdc317ff133eb8bba84c245eb8da/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/109a4775e16033b5d597a46c56e56307790ae4d656d222e889922bc5a44eaa68/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/4709ffc574d3616f2517d00877266e603ac0189153e7aa923becbf4a3b1d6de0/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/c8df92b5706cfb839c6e3ddd12da5cded56673903bd761fa8270c47ca316c34f/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/42406b199058723ee2836070d86da689cae16fe6e2e1ff4d49e6083056c6b2cc/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/1700321de1b7b8d83f068409827045b4e3fa4236d06a011aa7f7ec26874a0d7b/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/f05c4850d1a7b9fc9339c705e5e13756dc5e733960b1b7c340e39f0ff5f948f3/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/74076dd85c734610ad6bed6a9ca03ac7f4fdaae902ef3d64b11ee96d85685df2/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/38982dea8a84c2274678e7fff315c43440bb595dc85c6d02e43527335f3cd4c8/merged major:0 minor:98 fsType:overlay blockSize:0}] Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.742961 5232 manager.go:217] Machine: {Timestamp:2026-10-30 18:01:40.742463781 +0000 UTC m=+0.058884093 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3072751 MemoryCapacity:32812015616 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec23c59feae8cfc0ab014e430d2a0be8 SystemUUID:ec23c59f-eae8-cfc0-ab01-4e430d2a0be8 BootID:7b13398f-400a-49ec-9a05-0ad3464b240b Filesystems:[{Device:/var/lib/kubelet/pods/54bdc818-2b49-4f72-a168-6958864242fd/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1fe730a2abe2619857d71bec2084370e75929038ce5a74a4fa4b6d7113aab97/userdata/shm DeviceMajor:0 DeviceMinor:267 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f086c1b6538ec4dc3d3e05f8caba02cc55a8c3ae8aa5cbaf84b06c90dce9f03c/userdata/shm DeviceMajor:0 DeviceMinor:329 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9c7787af32f326b1a33fc8969a4fdee0caa75dfa0a53db231bc1d4dc60e524db/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/8754b620-b515-4ee4-9055-469a81126c9e/volumes/kubernetes.io~projected/kube-api-access-2nstb DeviceMajor:0 DeviceMinor:66 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:261 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/5e5fca56-5e6b-404d-84b2-4a876f5034cd/volumes/kubernetes.io~projected/kube-api-access-rsj7n DeviceMajor:0 DeviceMinor:61 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/17beb356-34fc-4c4a-97a4-72f00c649e7b/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:241 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/9dbc196724d7a247bbf69f1d92235f3cb08b6e2a86590d7a7b9e729289a7ddfc/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:259 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ed434e77d9eaa7fcb359f40da57c50494b5b63005a9e002add9d7ee20df0a13c/userdata/shm DeviceMajor:0 DeviceMinor:51 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b1a5bdb79ebf602be71a30fb617e07ad2b711b14427053b18a752e2bfd0a19ed/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7bd897e6-be5e-4d00-8faf-b2544c55a3f6/volumes/kubernetes.io~projected/kube-api-access-4rjmx DeviceMajor:0 DeviceMinor:202 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/31af1374-0d44-4636-bb2a-8e3836ac56b2/volumes/kubernetes.io~projected/kube-api-access-bhp2k DeviceMajor:0 DeviceMinor:204 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/a3c3fe6d-7fab-4cea-a04e-693b985ceb49/volumes/kubernetes.io~projected/kube-api-access-75xlp DeviceMajor:0 DeviceMinor:328 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/4627766d-20a1-46cd-baf9-cff7cc120679/volumes/kubernetes.io~projected/kube-api-access-q4xwn DeviceMajor:0 DeviceMinor:63 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:262 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/914c8c09-aa3d-4543-9d91-fa443a09b1ad/volumes/kubernetes.io~projected/kube-api-access-dtfcr DeviceMajor:0 DeviceMinor:60 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-142 DeviceMajor:0 DeviceMinor:142 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/29d1f2e2c17db36f7f56db2256ddbe2cae8e4cbc5dfce83649dae19e0c121e49/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-49 DeviceMajor:0 DeviceMinor:49 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/40dd1611-7ec9-411f-920d-e79df7273b90/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/b0259f4e815dae1a810bd384132879d3c437f9ee8cca15cdf54a11e49ce033e9/userdata/shm DeviceMajor:0 DeviceMinor:77 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-249 DeviceMajor:0 DeviceMinor:249 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abadb041-5d27-42f8-9cb9-35b78ec5f3bf/volumes/kubernetes.io~secret/client-ca-bundle DeviceMajor:0 DeviceMinor:282 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d0b2bad00761754a9924e522685d7e4559bac531c39d895e3d1caa2b9e78aa2c/userdata/shm DeviceMajor:0 DeviceMinor:285 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/17beb356-34fc-4c4a-97a4-72f00c649e7b/volumes/kubernetes.io~projected/kube-api-access-grpvw DeviceMajor:0 DeviceMinor:246 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-337 DeviceMajor:0 DeviceMinor:337 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-354 DeviceMajor:0 DeviceMinor:354 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a9dc368ba69895cdc26db91f28d278559f01fa3b3fbf328629be8e9be071331f/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5350ba5b6e2fc68a67e5d0f1eb3a7426e97c5a39f651ed171b95e2e062cf719e/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/abadb041-5d27-42f8-9cb9-35b78ec5f3bf/volumes/kubernetes.io~projected/kube-api-access-qx4m6 DeviceMajor:0 DeviceMinor:284 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-317 DeviceMajor:0 DeviceMinor:317 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/76146060-cfc6-4765-82a3-5c5c7601dfda/volumes/kubernetes.io~projected/kube-api-access-p496j DeviceMajor:0 DeviceMinor:67 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/8d711c2d-0597-4d83-a884-83e50d98378c/volumes/kubernetes.io~projected/kube-api-access-rbr88 DeviceMajor:0 DeviceMinor:59 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/bf6f41d2257c25cec09e37516e4d3dee5a780bc4239b38af7ec4540e8b434c4e/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/54bdc818-2b49-4f72-a168-6958864242fd/volumes/kubernetes.io~projected/kube-api-access-8clmh DeviceMajor:0 DeviceMinor:203 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-134 DeviceMajor:0 DeviceMinor:134 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3a6adbbde29a543a5e231d81cc1511870a1de772661753a445acbc5971f61c31/userdata/shm DeviceMajor:0 DeviceMinor:331 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/096c090ce580ec649397f1ed7348b86be8c7beb2f448f8db2a38365e1f676d04/userdata/shm DeviceMajor:0 DeviceMinor:333 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406007808 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6d458e1b-228c-4e8e-b66f-9688583b39db/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:264 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/b0df4b9d-89ce-4bb4-b759-68f1b499b52a/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:327 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-335 DeviceMajor:0 DeviceMinor:335 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0c1b111e1b4322013cec7a18f50d9149c78abb670cb2e693767430c964ffc492/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/76146060-cfc6-4765-82a3-5c5c7601dfda/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6d458e1b-228c-4e8e-b66f-9688583b39db/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/31af1374-0d44-4636-bb2a-8e3836ac56b2/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:45 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:253 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:259 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~projected/kube-api-access-75w7g DeviceMajor:0 DeviceMinor:266 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:259 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-341 DeviceMajor:0 DeviceMinor:341 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:260 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-269 DeviceMajor:0 DeviceMinor:269 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc/volumes/kubernetes.io~projected/kube-api-access-pkx2s DeviceMajor:0 DeviceMinor:65 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ffa54663ed165a1dbaec40fa9a7534778162d5b3fe9ee8e0ec42583267dbec8e/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:262 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-271 DeviceMajor:0 DeviceMinor:271 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/914c8c09-aa3d-4543-9d91-fa443a09b1ad/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:322 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:259 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-307 DeviceMajor:0 DeviceMinor:307 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-312 DeviceMajor:0 DeviceMinor:312 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-343 DeviceMajor:0 DeviceMinor:343 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:258 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/7bd897e6-be5e-4d00-8faf-b2544c55a3f6/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:201 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/17beb356-34fc-4c4a-97a4-72f00c649e7b/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:245 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/abadb041-5d27-42f8-9cb9-35b78ec5f3bf/volumes/kubernetes.io~secret/secret-metrics-server-client-certs DeviceMajor:0 DeviceMinor:278 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-289 DeviceMajor:0 DeviceMinor:289 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/03aabceb0285bf62c690b8cbb84759a7f4ceaaca8894efab90b4da611298d336/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:263 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/6d458e1b-228c-4e8e-b66f-9688583b39db/volumes/kubernetes.io~projected/kube-api-access-pm2mh DeviceMajor:0 DeviceMinor:64 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/8cdd06b5b549f9a7b3257e62bbbcf1aa9df0ab85b1dc5d5c9277216b2fb491a3/userdata/shm DeviceMajor:0 DeviceMinor:247 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:265 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-273 DeviceMajor:0 DeviceMinor:273 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-349 DeviceMajor:0 DeviceMinor:349 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/11311f90-d76a-42b5-b6a2-35c0b62bd222/volumes/kubernetes.io~projected/kube-api-access-bfmlm DeviceMajor:0 DeviceMinor:62 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-287 DeviceMajor:0 DeviceMinor:287 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/abadb041-5d27-42f8-9cb9-35b78ec5f3bf/volumes/kubernetes.io~secret/secret-metrics-server-tls DeviceMajor:0 DeviceMinor:283 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/76146060-cfc6-4765-82a3-5c5c7601dfda/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633416192 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f088090189782932f4518a0624192c82cea30262310c71044c8227e22c76e8a5/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:03aabceb0285bf6 MacAddress:42:91:7b:8a:89:07 Speed:10000 Mtu:8901} {Name:096c090ce580ec6 MacAddress:26:cd:58:af:47:dd Speed:10000 Mtu:8901} {Name:3a6adbbde29a543 MacAddress:96:20:51:0e:7e:88 Speed:10000 Mtu:8901} {Name:5350ba5b6e2fc68 MacAddress:56:ca:a7:5b:8d:ff Speed:10000 Mtu:8901} {Name:9dbc196724d7a24 MacAddress:1e:53:1f:96:13:b8 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ed:7c:0e:e7 Speed:0 Mtu:9001} {Name:br-int MacAddress:b6:70:f6:b2:65:36 Speed:0 Mtu:8901} {Name:d0b2bad00761754 MacAddress:42:da:cf:f1:b2:b1 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ed:7c:0e:e7 Speed:0 Mtu:9001} {Name:f086c1b6538ec4d MacAddress:7a:e8:df:55:36:6d Speed:10000 Mtu:8901} {Name:f1fe730a2abe261 MacAddress:3e:8b:5d:20:8f:ce Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ce:ed:74:a6:b1:af Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:85:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ae:4a:32:e4:f2:53 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812015616 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.743303 5232 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.743348 5232 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.743681 5232 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.743704 5232 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-106.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.743832 5232 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.743839 5232 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.743849 5232 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.743862 5232 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.743952 5232 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.744105 5232 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.744205 5232 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.744237 5232 kubelet.go:493] "Attempting to sync node with API server" Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.744246 5232 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.744256 5232 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.744266 5232 kubelet.go:405] "Adding apiserver pod source" Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.744274 5232 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.744955 5232 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 18:01:40.745773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.744965 5232 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:01:40.746818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.745927 5232 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 18:01:40.748398 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.748379 5232 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 18:01:40.748398 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.748395 5232 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 18:01:40.748583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.748503 5232 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 18:01:40.748755 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.748739 5232 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 18:01:40.748813 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.748780 5232 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 18:01:40.748905 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.748894 5232 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 18:01:40.748957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.748913 5232 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 18:01:40.748957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.748924 5232 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 18:01:40.748957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.748933 5232 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 18:01:40.748957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.748941 5232 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 18:01:40.748957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.748950 5232 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 18:01:40.749190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.748959 5232 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 18:01:40.749190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.748969 5232 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 18:01:40.749190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.748979 5232 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 18:01:40.749190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.748988 5232 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 18:01:40.749190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.749001 5232 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 18:01:40.749190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.749016 5232 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 18:01:40.749190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.749054 5232 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 18:01:40.749190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.749064 5232 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 18:01:40.752448 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.752432 5232 server.go:1263] "Started kubelet" Oct 30 18:01:40.753259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.752523 5232 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 18:01:40.753259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.752527 5232 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 18:01:40.753259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.753134 5232 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 18:01:40.753051 ip-10-0-141-106 systemd[1]: Started Kubernetes Kubelet. Oct 30 18:01:40.756035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.754776 5232 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 18:01:40.756035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.756000 5232 server.go:317] "Adding debug handlers to kubelet server" Oct 30 18:01:40.758609 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.758583 5232 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 18:01:40.758609 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.758602 5232 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 18:01:40.758780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.758660 5232 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 17:55:31 +0000 UTC" deadline="2028-05-09 16:06:06.647773673 +0000 UTC" Oct 30 18:01:40.758780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.758692 5232 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13366h4m25.889084287s" Oct 30 18:01:40.758780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.758694 5232 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 18:01:40.758780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.758708 5232 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 18:01:40.758780 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:01:40.758732 5232 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-106.ec2.internal\" not found" Oct 30 18:01:40.759018 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.758788 5232 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 18:01:40.759714 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.759610 5232 factory.go:55] Registering systemd factory Oct 30 18:01:40.759714 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.759633 5232 factory.go:223] Registration of the systemd container factory successfully Oct 30 18:01:40.760186 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.760168 5232 factory.go:153] Registering CRI-O factory Oct 30 18:01:40.760274 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.760191 5232 factory.go:223] Registration of the crio container factory successfully Oct 30 18:01:40.760274 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.760249 5232 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 18:01:40.760387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.760293 5232 factory.go:103] Registering Raw factory Oct 30 18:01:40.760387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.760308 5232 manager.go:1199] Started watching for new ooms in manager Oct 30 18:01:40.760723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.760704 5232 manager.go:319] Starting recovery of all containers Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.760942 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d458e1b-228c-4e8e-b66f-9688583b39db" volumeName="kubernetes.io/secret/6d458e1b-228c-4e8e-b66f-9688583b39db-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.760990 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76146060-cfc6-4765-82a3-5c5c7601dfda" volumeName="kubernetes.io/projected/76146060-cfc6-4765-82a3-5c5c7601dfda-kube-api-access-p496j" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761004 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8754b620-b515-4ee4-9055-469a81126c9e" volumeName="kubernetes.io/projected/8754b620-b515-4ee4-9055-469a81126c9e-kube-api-access-2nstb" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761016 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="914c8c09-aa3d-4543-9d91-fa443a09b1ad" volumeName="kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761027 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb82def8-d679-4f36-8d87-b9b23865e3de" volumeName="kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-config-volume" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761040 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4627766d-20a1-46cd-baf9-cff7cc120679" volumeName="kubernetes.io/projected/4627766d-20a1-46cd-baf9-cff7cc120679-kube-api-access-q4xwn" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761051 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76146060-cfc6-4765-82a3-5c5c7601dfda" volumeName="kubernetes.io/empty-dir/76146060-cfc6-4765-82a3-5c5c7601dfda-tmp" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761063 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8754b620-b515-4ee4-9055-469a81126c9e" volumeName="kubernetes.io/configmap/8754b620-b515-4ee4-9055-469a81126c9e-multus-daemon-config" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761078 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="914c8c09-aa3d-4543-9d91-fa443a09b1ad" volumeName="kubernetes.io/projected/914c8c09-aa3d-4543-9d91-fa443a09b1ad-kube-api-access-dtfcr" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761089 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abadb041-5d27-42f8-9cb9-35b78ec5f3bf" volumeName="kubernetes.io/secret/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-client-ca-bundle" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761100 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11311f90-d76a-42b5-b6a2-35c0b62bd222" volumeName="kubernetes.io/projected/11311f90-d76a-42b5-b6a2-35c0b62bd222-kube-api-access-bfmlm" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761110 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abadb041-5d27-42f8-9cb9-35b78ec5f3bf" volumeName="kubernetes.io/projected/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-kube-api-access-qx4m6" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761121 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb82def8-d679-4f36-8d87-b9b23865e3de" volumeName="kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-main-db" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761133 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bd897e6-be5e-4d00-8faf-b2544c55a3f6" volumeName="kubernetes.io/projected/7bd897e6-be5e-4d00-8faf-b2544c55a3f6-kube-api-access-4rjmx" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761147 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abadb041-5d27-42f8-9cb9-35b78ec5f3bf" volumeName="kubernetes.io/configmap/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-metrics-server-audit-profiles" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761159 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb82def8-d679-4f36-8d87-b9b23865e3de" volumeName="kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761171 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17beb356-34fc-4c4a-97a4-72f00c649e7b" volumeName="kubernetes.io/configmap/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761190 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54bdc818-2b49-4f72-a168-6958864242fd" volumeName="kubernetes.io/empty-dir/54bdc818-2b49-4f72-a168-6958864242fd-tmp-dir" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761203 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc" volumeName="kubernetes.io/projected/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-kube-api-access-pkx2s" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761216 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e5fca56-5e6b-404d-84b2-4a876f5034cd" volumeName="kubernetes.io/projected/5e5fca56-5e6b-404d-84b2-4a876f5034cd-kube-api-access-rsj7n" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761229 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d458e1b-228c-4e8e-b66f-9688583b39db" volumeName="kubernetes.io/configmap/6d458e1b-228c-4e8e-b66f-9688583b39db-env-overrides" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761242 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8754b620-b515-4ee4-9055-469a81126c9e" volumeName="kubernetes.io/configmap/8754b620-b515-4ee4-9055-469a81126c9e-cni-binary-copy" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761254 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abadb041-5d27-42f8-9cb9-35b78ec5f3bf" volumeName="kubernetes.io/secret/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-secret-metrics-server-client-certs" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761266 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b0df4b9d-89ce-4bb4-b759-68f1b499b52a" volumeName="kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761278 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17beb356-34fc-4c4a-97a4-72f00c649e7b" volumeName="kubernetes.io/projected/17beb356-34fc-4c4a-97a4-72f00c649e7b-kube-api-access-grpvw" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761291 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d458e1b-228c-4e8e-b66f-9688583b39db" volumeName="kubernetes.io/configmap/6d458e1b-228c-4e8e-b66f-9688583b39db-ovnkube-script-lib" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761303 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb82def8-d679-4f36-8d87-b9b23865e3de" volumeName="kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-config-out" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761315 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb82def8-d679-4f36-8d87-b9b23865e3de" volumeName="kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761347 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="11311f90-d76a-42b5-b6a2-35c0b62bd222" volumeName="kubernetes.io/configmap/11311f90-d76a-42b5-b6a2-35c0b62bd222-serviceca" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761391 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17beb356-34fc-4c4a-97a4-72f00c649e7b" volumeName="kubernetes.io/configmap/17beb356-34fc-4c4a-97a4-72f00c649e7b-metrics-client-ca" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761405 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40dd1611-7ec9-411f-920d-e79df7273b90" volumeName="kubernetes.io/configmap/40dd1611-7ec9-411f-920d-e79df7273b90-konnectivity-ca" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761419 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54bdc818-2b49-4f72-a168-6958864242fd" volumeName="kubernetes.io/secret/54bdc818-2b49-4f72-a168-6958864242fd-metrics-tls" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761433 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="7bd897e6-be5e-4d00-8faf-b2544c55a3f6" volumeName="kubernetes.io/secret/7bd897e6-be5e-4d00-8faf-b2544c55a3f6-cert" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761447 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb82def8-d679-4f36-8d87-b9b23865e3de" volumeName="kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-kube-api-access-75w7g" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761459 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17beb356-34fc-4c4a-97a4-72f00c649e7b" volumeName="kubernetes.io/empty-dir/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-textfile" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761471 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17beb356-34fc-4c4a-97a4-72f00c649e7b" volumeName="kubernetes.io/secret/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761484 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abadb041-5d27-42f8-9cb9-35b78ec5f3bf" volumeName="kubernetes.io/configmap/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-configmap-kubelet-serving-ca-bundle" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761497 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abadb041-5d27-42f8-9cb9-35b78ec5f3bf" volumeName="kubernetes.io/empty-dir/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-audit-log" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761564 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb82def8-d679-4f36-8d87-b9b23865e3de" volumeName="kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-cluster-tls-config" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761580 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc" volumeName="kubernetes.io/configmap/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761591 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d458e1b-228c-4e8e-b66f-9688583b39db" volumeName="kubernetes.io/projected/6d458e1b-228c-4e8e-b66f-9688583b39db-kube-api-access-pm2mh" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761603 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a3c3fe6d-7fab-4cea-a04e-693b985ceb49" volumeName="kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761615 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb82def8-d679-4f36-8d87-b9b23865e3de" volumeName="kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-tls-assets" seLinuxMountContext="" Oct 30 18:01:40.761597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761628 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6d458e1b-228c-4e8e-b66f-9688583b39db" volumeName="kubernetes.io/configmap/6d458e1b-228c-4e8e-b66f-9688583b39db-ovnkube-config" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761639 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31af1374-0d44-4636-bb2a-8e3836ac56b2" volumeName="kubernetes.io/configmap/31af1374-0d44-4636-bb2a-8e3836ac56b2-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761651 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54bdc818-2b49-4f72-a168-6958864242fd" volumeName="kubernetes.io/projected/54bdc818-2b49-4f72-a168-6958864242fd-kube-api-access-8clmh" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761661 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc" volumeName="kubernetes.io/configmap/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-cni-binary-copy" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761673 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d711c2d-0597-4d83-a884-83e50d98378c" volumeName="kubernetes.io/empty-dir/8d711c2d-0597-4d83-a884-83e50d98378c-tmp-dir" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761683 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb82def8-d679-4f36-8d87-b9b23865e3de" volumeName="kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761706 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54bdc818-2b49-4f72-a168-6958864242fd" volumeName="kubernetes.io/configmap/54bdc818-2b49-4f72-a168-6958864242fd-config-volume" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761716 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="abadb041-5d27-42f8-9cb9-35b78ec5f3bf" volumeName="kubernetes.io/secret/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-secret-metrics-server-tls" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761727 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31af1374-0d44-4636-bb2a-8e3836ac56b2" volumeName="kubernetes.io/projected/31af1374-0d44-4636-bb2a-8e3836ac56b2-kube-api-access-bhp2k" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761739 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc" volumeName="kubernetes.io/configmap/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761750 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5e5fca56-5e6b-404d-84b2-4a876f5034cd" volumeName="kubernetes.io/configmap/5e5fca56-5e6b-404d-84b2-4a876f5034cd-iptables-alerter-script" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761763 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="76146060-cfc6-4765-82a3-5c5c7601dfda" volumeName="kubernetes.io/empty-dir/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-tuned" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761777 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8d711c2d-0597-4d83-a884-83e50d98378c" volumeName="kubernetes.io/projected/8d711c2d-0597-4d83-a884-83e50d98378c-kube-api-access-rbr88" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761800 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb82def8-d679-4f36-8d87-b9b23865e3de" volumeName="kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761814 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31af1374-0d44-4636-bb2a-8e3836ac56b2" volumeName="kubernetes.io/empty-dir/31af1374-0d44-4636-bb2a-8e3836ac56b2-data-volume" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761826 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb82def8-d679-4f36-8d87-b9b23865e3de" volumeName="kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-metrics-client-ca" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761838 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb82def8-d679-4f36-8d87-b9b23865e3de" volumeName="kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-web-config" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761850 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fb82def8-d679-4f36-8d87-b9b23865e3de" volumeName="kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761865 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="17beb356-34fc-4c4a-97a4-72f00c649e7b" volumeName="kubernetes.io/secret/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-tls" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761877 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="31af1374-0d44-4636-bb2a-8e3836ac56b2" volumeName="kubernetes.io/secret/31af1374-0d44-4636-bb2a-8e3836ac56b2-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761889 5232 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="40dd1611-7ec9-411f-920d-e79df7273b90" volumeName="kubernetes.io/secret/40dd1611-7ec9-411f-920d-e79df7273b90-agent-certs" seLinuxMountContext="" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761902 5232 reconstruct.go:97] "Volume reconstruction finished" Oct 30 18:01:40.763060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.761910 5232 reconciler.go:29] "Reconciler: start to sync state" Oct 30 18:01:40.792706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.792642 5232 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 18:01:40.794606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.794559 5232 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 18:01:40.794845 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.794615 5232 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 18:01:40.794845 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.794647 5232 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 18:01:40.794845 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:01:40.794703 5232 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 18:01:40.803585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.803554 5232 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:01:40.803705 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.803688 5232 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 18:01:40.804529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.804487 5232 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:01:40.808466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.808439 5232 generic.go:356] "Generic (PLEG): container finished" podID="2ace45daccc39fb3be3eb84e89404869" containerID="0540e17ef165c43ba26e209eba50a0d1c6ea585af736bd92cade2103b1424970" exitCode=0 Oct 30 18:01:40.812139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.812117 5232 generic.go:356] "Generic (PLEG): container finished" podID="17beb356-34fc-4c4a-97a4-72f00c649e7b" containerID="a6cc9561f4d7d29ff1bcf347db6d5dece554bd6d23f9c6e825bf1e10d9075ae3" exitCode=0 Oct 30 18:01:40.827266 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.827237 5232 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:01:40.828063 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.828039 5232 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 18:01:40.835631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.835605 5232 generic.go:356] "Generic (PLEG): container finished" podID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerID="eae7e933f2eb2fa8270d189d2f72af5847ed46a7ac5e26e71695065c7a25e4fd" exitCode=0 Oct 30 18:01:40.839206 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.839174 5232 generic.go:356] "Generic (PLEG): container finished" podID="54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc" containerID="a62e035b8e8571735eec3afbc03b6d8b970d1a2cd2ada35798350403d5c6a40b" exitCode=0 Oct 30 18:01:40.839206 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.839206 5232 generic.go:356] "Generic (PLEG): container finished" podID="54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc" containerID="b70c97d6538689d0db1b7d30e1730076599dda21a5702e1639cef84c2f53b41f" exitCode=0 Oct 30 18:01:40.839355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.839216 5232 generic.go:356] "Generic (PLEG): container finished" podID="54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc" containerID="13b762ec051cb6abf2b396b35e54a7a98e917f1725d472c8009ea76cfb6634eb" exitCode=0 Oct 30 18:01:40.839355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.839225 5232 generic.go:356] "Generic (PLEG): container finished" podID="54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc" containerID="588ea41a3006a7398c9a8fec71a1f1c520724c34485b355bb719aa79a10672d4" exitCode=0 Oct 30 18:01:40.839355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.839234 5232 generic.go:356] "Generic (PLEG): container finished" podID="54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc" containerID="6308d0b4b938a0996825585cfde44cf56e5f4f8466d38372185a14fda6c78063" exitCode=0 Oct 30 18:01:40.839355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.839243 5232 generic.go:356] "Generic (PLEG): container finished" podID="54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc" containerID="42b94f51956653b1208d881b8325bf23a1bb7c05bd264ede76db1ee61a89c145" exitCode=0 Oct 30 18:01:40.851214 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.851188 5232 manager.go:324] Recovery completed Oct 30 18:01:40.870128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.870058 5232 cpu_manager.go:226] "Starting" policy="none" Oct 30 18:01:40.870128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.870077 5232 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 18:01:40.870128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.870099 5232 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 18:01:40.870311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.870232 5232 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 18:01:40.870311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.870242 5232 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 18:01:40.870311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.870260 5232 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 18:01:40.870311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.870265 5232 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 18:01:40.870311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.870270 5232 policy_none.go:50] "Start" Oct 30 18:01:40.870311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.870278 5232 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 18:01:40.870311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.870287 5232 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 18:01:40.870550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.870411 5232 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 18:01:40.870550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.870418 5232 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 18:01:40.870550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.870423 5232 policy_none.go:44] "Start" Oct 30 18:01:40.875888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.875853 5232 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 18:01:40.878490 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.878477 5232 manager.go:341] "Starting Device Plugin manager" Oct 30 18:01:40.879050 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:01:40.878519 5232 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 18:01:40.879114 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.879053 5232 server.go:87] "Starting device plugin registration server" Oct 30 18:01:40.879317 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.879306 5232 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 18:01:40.879351 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.879318 5232 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 18:01:40.879430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.879408 5232 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 18:01:40.879569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.879538 5232 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 18:01:40.879643 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.879578 5232 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 18:01:40.879643 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.879582 5232 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 18:01:40.879731 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.879644 5232 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T18:01:40.879566798Z","UUID":"56c5de65-b2f9-44ba-885a-2fcfb1ff02c9","Handler":null,"Name":"","Endpoint":""} Oct 30 18:01:40.880788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.880746 5232 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 18:01:40.880788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.880775 5232 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 18:01:40.881402 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:01:40.881383 5232 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 18:01:40.894804 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.894771 5232 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal"] Oct 30 18:01:40.895290 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.895259 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" event={"ID":"2ace45daccc39fb3be3eb84e89404869","Type":"ContainerStarted","Data":"80c9a70c4019113c14738128c5a4182c8d7972af6c3d3df16972cf3c0276c9b3"} Oct 30 18:01:40.895373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.895298 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" event={"ID":"2ace45daccc39fb3be3eb84e89404869","Type":"ContainerDied","Data":"0540e17ef165c43ba26e209eba50a0d1c6ea585af736bd92cade2103b1424970"} Oct 30 18:01:40.895373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.895325 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" event={"ID":"2ace45daccc39fb3be3eb84e89404869","Type":"ContainerStarted","Data":"ed434e77d9eaa7fcb359f40da57c50494b5b63005a9e002add9d7ee20df0a13c"} Oct 30 18:01:40.895373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.895344 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" event={"ID":"1bdd65ea12e9d18ae4c0eda6db4fb016","Type":"ContainerStarted","Data":"107bd39ba7e2b3ea22fb40adf5d4235c8c7e9ae6ba148145313783c0f0558545"} Oct 30 18:01:40.895373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.895352 5232 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:01:40.895373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.895360 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" event={"ID":"1bdd65ea12e9d18ae4c0eda6db4fb016","Type":"ContainerStarted","Data":"0c1b111e1b4322013cec7a18f50d9149c78abb670cb2e693767430c964ffc492"} Oct 30 18:01:40.895608 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.895498 5232 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Oct 30 18:01:40.919422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.919400 5232 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:01:40.919543 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:01:40.919449 5232 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-106.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Oct 30 18:01:40.919822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.919807 5232 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:01:40.919877 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:01:40.919846 5232 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:01:40.979961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.979937 5232 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 18:01:40.981820 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.981801 5232 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientMemory" Oct 30 18:01:40.981922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.981837 5232 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasNoDiskPressure" Oct 30 18:01:40.981922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.981851 5232 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-106.ec2.internal" event="NodeHasSufficientPID" Oct 30 18:01:40.981922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:40.981896 5232 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-106.ec2.internal" Oct 30 18:01:41.003822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.003800 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ace45daccc39fb3be3eb84e89404869-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" (UID: \"2ace45daccc39fb3be3eb84e89404869\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:01:41.003924 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.003827 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ace45daccc39fb3be3eb84e89404869-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" (UID: \"2ace45daccc39fb3be3eb84e89404869\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:01:41.003924 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.003845 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1bdd65ea12e9d18ae4c0eda6db4fb016-config\") pod \"kube-apiserver-proxy-ip-10-0-141-106.ec2.internal\" (UID: \"1bdd65ea12e9d18ae4c0eda6db4fb016\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Oct 30 18:01:41.025436 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.025415 5232 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-106.ec2.internal" Oct 30 18:01:41.025531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.025466 5232 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-106.ec2.internal" Oct 30 18:01:41.104813 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.104775 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1bdd65ea12e9d18ae4c0eda6db4fb016-config\") pod \"kube-apiserver-proxy-ip-10-0-141-106.ec2.internal\" (UID: \"1bdd65ea12e9d18ae4c0eda6db4fb016\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Oct 30 18:01:41.104813 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.104811 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ace45daccc39fb3be3eb84e89404869-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" (UID: \"2ace45daccc39fb3be3eb84e89404869\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:01:41.105008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.104832 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ace45daccc39fb3be3eb84e89404869-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" (UID: \"2ace45daccc39fb3be3eb84e89404869\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:01:41.105008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.104892 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2ace45daccc39fb3be3eb84e89404869-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" (UID: \"2ace45daccc39fb3be3eb84e89404869\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:01:41.105008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.104911 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2ace45daccc39fb3be3eb84e89404869-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" (UID: \"2ace45daccc39fb3be3eb84e89404869\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:01:41.105008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.104893 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/1bdd65ea12e9d18ae4c0eda6db4fb016-config\") pod \"kube-apiserver-proxy-ip-10-0-141-106.ec2.internal\" (UID: \"1bdd65ea12e9d18ae4c0eda6db4fb016\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Oct 30 18:01:41.745201 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.745161 5232 apiserver.go:52] "Watching apiserver" Oct 30 18:01:41.785399 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.785369 5232 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 18:01:41.785964 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.785943 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal","openshift-dns/node-resolver-td6vr","openshift-multus/multus-additional-cni-plugins-mw76k","openshift-monitoring/node-exporter-5zplp","openshift-multus/network-metrics-daemon-g8h9t","openshift-dns/dns-default-st6k6","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal","openshift-network-diagnostics/network-check-target-94r4x","kube-system/konnectivity-agent-zjlpz","openshift-network-operator/iptables-alerter-t7sqv","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5","openshift-monitoring/metrics-server-866f94cc5d-rv8z2","openshift-ingress-canary/ingress-canary-kqvrz","openshift-cluster-node-tuning-operator/tuned-k8684","openshift-multus/multus-9bq58","openshift-ovn-kubernetes/ovnkube-node-m556r","openshift-insights/insights-runtime-extractor-b9kd2","kube-system/global-pull-secret-syncer-brgpf","openshift-image-registry/node-ca-h6lp9","openshift-monitoring/alertmanager-main-0"] Oct 30 18:01:41.801816 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.801787 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 18:01:41.802006 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.801893 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 18:01:41.802006 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.801992 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:01:41.802006 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.801994 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 18:01:41.802151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.802062 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-b72w5\"" Oct 30 18:01:41.802151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.802079 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 18:01:41.802151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.802097 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 18:01:41.802151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.802118 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 18:01:41.802327 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.802164 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-hdw4l\"" Oct 30 18:01:41.802327 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.802183 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 18:01:41.802327 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.802205 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 18:01:41.802327 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.802234 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 18:01:41.815424 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.815400 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-dzg75\"" Oct 30 18:01:41.815584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.815454 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-tlc5r\"" Oct 30 18:01:41.815793 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.815780 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-qrbw2\"" Oct 30 18:01:41.815922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.815906 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-qdst5\"" Oct 30 18:01:41.815987 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.815927 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 18:01:41.815987 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.815930 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 18:01:41.815987 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.815974 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 18:01:41.816094 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.815989 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 18:01:41.816094 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.815989 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 18:01:41.816155 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.816125 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 18:01:41.816270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.816252 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-qfvjp\"" Oct 30 18:01:41.816270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.816253 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 18:01:41.816436 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.816330 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 18:01:41.816436 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.816343 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 18:01:41.816436 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.816352 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 18:01:41.816667 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.816652 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 18:01:41.816736 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.816686 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 18:01:41.816736 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.816716 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:01:41.816835 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.816737 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 18:01:41.816835 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.816751 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 18:01:41.816835 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.816767 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 18:01:41.816835 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.816804 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 18:01:41.817062 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817046 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 18:01:41.817104 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817091 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 18:01:41.817257 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817233 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 18:01:41.817257 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817232 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fqgf4\"" Oct 30 18:01:41.817257 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817237 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 18:01:41.817447 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817269 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 18:01:41.817447 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817290 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" Oct 30 18:01:41.817447 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817333 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" Oct 30 18:01:41.817447 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817423 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-ljv2v\"" Oct 30 18:01:41.817447 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817434 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 18:01:41.817619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817433 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 18:01:41.817619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817471 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 18:01:41.817619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817548 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-l5dm7\"" Oct 30 18:01:41.817738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817696 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 18:01:41.817738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817706 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-d6j24\"" Oct 30 18:01:41.817738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817718 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8z8qv\"" Oct 30 18:01:41.817826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817736 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 18:01:41.817826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817744 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 18:01:41.817826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.817738 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-hj78h\"" Oct 30 18:01:41.826859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.826843 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 18:01:41.826859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.826856 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-wc69q\"" Oct 30 18:01:41.826986 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.826944 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 18:01:41.826986 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.826960 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-qzhdb\"" Oct 30 18:01:41.827074 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.826991 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 18:01:41.827074 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827018 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 18:01:41.827299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827279 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 18:01:41.827299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827291 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 18:01:41.827299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827284 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 18:01:41.827481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827307 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-4js9k\"" Oct 30 18:01:41.827481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827314 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 18:01:41.827481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827331 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 18:01:41.827481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827344 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" Oct 30 18:01:41.827481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827354 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 18:01:41.827481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827393 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" Oct 30 18:01:41.827481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827445 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 18:01:41.827481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827452 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 18:01:41.827481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827465 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 18:01:41.827481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827467 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 18:01:41.827837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827545 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-mj45s\"" Oct 30 18:01:41.827837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827445 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 18:01:41.827837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827486 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 18:01:41.827837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827632 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-khr2s\"" Oct 30 18:01:41.827837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.827785 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-c0frpd4sn3mil\"" Oct 30 18:01:41.829306 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.829291 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 18:01:41.847469 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.847452 5232 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Oct 30 18:01:41.847670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.847655 5232 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:01:41.860480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.860461 5232 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 18:01:41.870270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.870251 5232 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:01:41.870362 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:01:41.870296 5232 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-106.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-106.ec2.internal" Oct 30 18:01:41.870592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.870577 5232 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 18:01:41.870647 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:01:41.870628 5232 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-106.ec2.internal" Oct 30 18:01:41.908712 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.908688 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-var-lib-cni-multus\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.908831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.908719 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nstb\" (UniqueName: \"kubernetes.io/projected/8754b620-b515-4ee4-9055-469a81126c9e-kube-api-access-2nstb\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.908831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.908737 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:41.908831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.908752 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs\") pod \"network-metrics-daemon-g8h9t\" (UID: \"914c8c09-aa3d-4543-9d91-fa443a09b1ad\") " pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:01:41.908971 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.908825 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8754b620-b515-4ee4-9055-469a81126c9e-multus-daemon-config\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.908971 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.908920 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e5fca56-5e6b-404d-84b2-4a876f5034cd-host-slash\") pod \"iptables-alerter-t7sqv\" (UID: \"5e5fca56-5e6b-404d-84b2-4a876f5034cd\") " pod="openshift-network-operator/iptables-alerter-t7sqv" Oct 30 18:01:41.909354 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.909327 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:01:41.909449 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.909430 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-run-multus-certs\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.909539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.909452 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/914c8c09-aa3d-4543-9d91-fa443a09b1ad-metrics-certs\") pod \"network-metrics-daemon-g8h9t\" (UID: \"914c8c09-aa3d-4543-9d91-fa443a09b1ad\") " pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:01:41.909539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.909473 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-sysctl-conf\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:41.909539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.909501 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:41.909692 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.909561 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-textfile\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:41.909692 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.909568 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:01:41.909692 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.909592 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8clmh\" (UniqueName: \"kubernetes.io/projected/54bdc818-2b49-4f72-a168-6958864242fd-kube-api-access-8clmh\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:41.909692 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.909626 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.909692 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.909659 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/8754b620-b515-4ee4-9055-469a81126c9e-multus-daemon-config\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.909692 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.909678 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-system-cni-dir\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.909692 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.909688 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-textfile\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:41.909989 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.909719 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-multus-conf-dir\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.909989 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.909777 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-wtmp\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:41.910086 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910019 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.910086 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910025 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-sysctl-d\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:41.910086 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910075 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bd897e6-be5e-4d00-8faf-b2544c55a3f6-cert\") pod \"ingress-canary-kqvrz\" (UID: \"7bd897e6-be5e-4d00-8faf-b2544c55a3f6\") " pod="openshift-ingress-canary/ingress-canary-kqvrz" Oct 30 18:01:41.910227 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910104 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17beb356-34fc-4c4a-97a4-72f00c649e7b-metrics-client-ca\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:41.910227 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910139 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.910227 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910194 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-lib-modules\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:41.910365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910227 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-web-config\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.910365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910259 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-sys\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:41.910365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910289 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-cnibin\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:01:41.910365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910333 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.910365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910337 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.910627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910367 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/7bd897e6-be5e-4d00-8faf-b2544c55a3f6-cert\") pod \"ingress-canary-kqvrz\" (UID: \"7bd897e6-be5e-4d00-8faf-b2544c55a3f6\") " pod="openshift-ingress-canary/ingress-canary-kqvrz" Oct 30 18:01:41.910627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910396 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-run-openvswitch\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.910627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910446 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-cni-netd\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.910627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910479 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-slash\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.910627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910542 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsj7n\" (UniqueName: \"kubernetes.io/projected/5e5fca56-5e6b-404d-84b2-4a876f5034cd-kube-api-access-rsj7n\") pod \"iptables-alerter-t7sqv\" (UID: \"5e5fca56-5e6b-404d-84b2-4a876f5034cd\") " pod="openshift-network-operator/iptables-alerter-t7sqv" Oct 30 18:01:41.910627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910575 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-socket-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:41.910627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910579 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-tls-assets\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.910627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910602 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/17beb356-34fc-4c4a-97a4-72f00c649e7b-metrics-client-ca\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:41.910627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910617 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-run-k8s-cni-cncf-io\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.911017 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910856 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.911017 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910870 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-web-config\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.911017 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.910953 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-log-socket\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.911017 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911008 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54bdc818-2b49-4f72-a168-6958864242fd-config-volume\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:41.911185 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911070 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-cnibin\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.911185 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911120 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-client-ca-bundle\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:41.911185 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911156 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-run-ovn-kubernetes\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.911355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911208 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:41.911355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911231 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-kubernetes\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:41.911355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911262 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-host\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:41.911355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911275 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/54bdc818-2b49-4f72-a168-6958864242fd-config-volume\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:41.911355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911292 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-kubelet-config\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:41.911355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911347 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-sys-fs\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:41.911626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911370 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-kubelet\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.911626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911401 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d458e1b-228c-4e8e-b66f-9688583b39db-ovn-node-metrics-cert\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.911626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911428 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54bdc818-2b49-4f72-a168-6958864242fd-metrics-tls\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:41.911626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911459 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-systemd\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:41.911626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911485 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11311f90-d76a-42b5-b6a2-35c0b62bd222-host\") pod \"node-ca-h6lp9\" (UID: \"11311f90-d76a-42b5-b6a2-35c0b62bd222\") " pod="openshift-image-registry/node-ca-h6lp9" Oct 30 18:01:41.911626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911539 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.911626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911543 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:41.911626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911581 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grpvw\" (UniqueName: \"kubernetes.io/projected/17beb356-34fc-4c4a-97a4-72f00c649e7b-kube-api-access-grpvw\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:41.911939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.911890 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-client-ca-bundle\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:41.912066 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.912046 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.912162 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.912149 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-etc-openvswitch\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.912250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.912238 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-node-log\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.912324 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.912198 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/6d458e1b-228c-4e8e-b66f-9688583b39db-ovn-node-metrics-cert\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.912445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.912429 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-device-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:41.912571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.912556 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4xwn\" (UniqueName: \"kubernetes.io/projected/4627766d-20a1-46cd-baf9-cff7cc120679-kube-api-access-q4xwn\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:41.912681 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.912669 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8754b620-b515-4ee4-9055-469a81126c9e-cni-binary-copy\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.912774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.912761 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.912871 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.912859 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-tuned\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:41.912961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.912949 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-system-cni-dir\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:01:41.913044 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.913025 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/54bdc818-2b49-4f72-a168-6958864242fd-metrics-tls\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:41.913097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.912777 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.913197 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.913180 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx4m6\" (UniqueName: \"kubernetes.io/projected/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-kube-api-access-qx4m6\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:41.913250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.913234 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dtfcr\" (UniqueName: \"kubernetes.io/projected/914c8c09-aa3d-4543-9d91-fa443a09b1ad-kube-api-access-dtfcr\") pod \"network-metrics-daemon-g8h9t\" (UID: \"914c8c09-aa3d-4543-9d91-fa443a09b1ad\") " pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:01:41.913294 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.913262 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-var-lib-cni-bin\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.913337 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.913293 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-config-volume\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.913337 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.913322 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-cni-bin\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.913418 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.913354 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5e5fca56-5e6b-404d-84b2-4a876f5034cd-iptables-alerter-script\") pod \"iptables-alerter-t7sqv\" (UID: \"5e5fca56-5e6b-404d-84b2-4a876f5034cd\") " pod="openshift-network-operator/iptables-alerter-t7sqv" Oct 30 18:01:41.913487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.913471 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-systemd-units\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.913564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.913538 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76146060-cfc6-4765-82a3-5c5c7601dfda-tmp\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:41.913616 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.913567 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d711c2d-0597-4d83-a884-83e50d98378c-hosts-file\") pod \"node-resolver-td6vr\" (UID: \"8d711c2d-0597-4d83-a884-83e50d98378c\") " pod="openshift-dns/node-resolver-td6vr" Oct 30 18:01:41.913616 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.913600 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-secret-metrics-server-tls\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:41.913720 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.913632 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17beb356-34fc-4c4a-97a4-72f00c649e7b-sys\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:41.913720 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.913685 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-config-volume\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.913720 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.913698 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-accelerators-collector-config\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:41.913720 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.913713 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.913861 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.913746 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/76146060-cfc6-4765-82a3-5c5c7601dfda-tmp\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:41.913901 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.913887 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.913955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.913932 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-tuned\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:41.914495 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.914468 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d458e1b-228c-4e8e-b66f-9688583b39db-ovnkube-config\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.914604 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.914545 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-secret-metrics-server-tls\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:41.914672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.914604 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8754b620-b515-4ee4-9055-469a81126c9e-cni-binary-copy\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.914672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.914615 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/5e5fca56-5e6b-404d-84b2-4a876f5034cd-iptables-alerter-script\") pod \"iptables-alerter-t7sqv\" (UID: \"5e5fca56-5e6b-404d-84b2-4a876f5034cd\") " pod="openshift-network-operator/iptables-alerter-t7sqv" Oct 30 18:01:41.914672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.914616 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31af1374-0d44-4636-bb2a-8e3836ac56b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:41.914821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.914756 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/6d458e1b-228c-4e8e-b66f-9688583b39db-ovnkube-config\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.914821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.914812 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.914912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.914850 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40dd1611-7ec9-411f-920d-e79df7273b90-konnectivity-ca\") pod \"konnectivity-agent-zjlpz\" (UID: \"40dd1611-7ec9-411f-920d-e79df7273b90\") " pod="kube-system/konnectivity-agent-zjlpz" Oct 30 18:01:41.914912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.914850 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/31af1374-0d44-4636-bb2a-8e3836ac56b2-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:41.914912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.914880 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-modprobe-d\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:41.915057 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.914983 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-secret-metrics-server-client-certs\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:41.915057 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915015 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:41.915057 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915019 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/40dd1611-7ec9-411f-920d-e79df7273b90-konnectivity-ca\") pod \"konnectivity-agent-zjlpz\" (UID: \"40dd1611-7ec9-411f-920d-e79df7273b90\") " pod="kube-system/konnectivity-agent-zjlpz" Oct 30 18:01:41.915189 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915076 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rjmx\" (UniqueName: \"kubernetes.io/projected/7bd897e6-be5e-4d00-8faf-b2544c55a3f6-kube-api-access-4rjmx\") pod \"ingress-canary-kqvrz\" (UID: \"7bd897e6-be5e-4d00-8faf-b2544c55a3f6\") " pod="openshift-ingress-canary/ingress-canary-kqvrz" Oct 30 18:01:41.915189 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915108 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-run\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:41.915189 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915172 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:01:41.915316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915214 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54bdc818-2b49-4f72-a168-6958864242fd-tmp-dir\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:41.915316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915237 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-secret-metrics-server-client-certs\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:41.915316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915270 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-dbus\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:41.915316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915308 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-var-lib-openvswitch\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.915531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915326 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-original-pull-secret\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:41.915531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915331 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54bdc818-2b49-4f72-a168-6958864242fd-tmp-dir\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:41.915531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915347 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.915531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915387 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-var-lib-kubelet\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:41.915531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915439 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.915778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915548 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-metrics-server-audit-profiles\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:41.915778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915595 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11311f90-d76a-42b5-b6a2-35c0b62bd222-serviceca\") pod \"node-ca-h6lp9\" (UID: \"11311f90-d76a-42b5-b6a2-35c0b62bd222\") " pod="openshift-image-registry/node-ca-h6lp9" Oct 30 18:01:41.915778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915679 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.915778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915685 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-config-out\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.915778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915734 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75xlp\" (UniqueName: \"kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp\") pod \"network-check-target-94r4x\" (UID: \"a3c3fe6d-7fab-4cea-a04e-693b985ceb49\") " pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:01:41.915778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915765 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:41.916059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915822 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d458e1b-228c-4e8e-b66f-9688583b39db-ovnkube-script-lib\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.916059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915835 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-config-out\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.916059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915853 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-metrics-server-audit-profiles\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:41.916059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915882 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pm2mh\" (UniqueName: \"kubernetes.io/projected/6d458e1b-228c-4e8e-b66f-9688583b39db-kube-api-access-pm2mh\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.916059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915926 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:01:41.916059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915945 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-audit-log\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:41.916059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915968 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:41.916059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.915986 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-multus-socket-dir-parent\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.916059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916032 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75w7g\" (UniqueName: \"kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-kube-api-access-75w7g\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.916059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916037 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-audit-log\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:41.916435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916105 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-run-netns\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.916435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916126 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:01:41.916435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916137 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/6d458e1b-228c-4e8e-b66f-9688583b39db-ovnkube-script-lib\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.916435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916146 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31af1374-0d44-4636-bb2a-8e3836ac56b2-crio-socket\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:41.916435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916176 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bhp2k\" (UniqueName: \"kubernetes.io/projected/31af1374-0d44-4636-bb2a-8e3836ac56b2-kube-api-access-bhp2k\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:41.916435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916198 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40dd1611-7ec9-411f-920d-e79df7273b90-agent-certs\") pod \"konnectivity-agent-zjlpz\" (UID: \"40dd1611-7ec9-411f-920d-e79df7273b90\") " pod="kube-system/konnectivity-agent-zjlpz" Oct 30 18:01:41.916435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916279 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.916435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916264 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/11311f90-d76a-42b5-b6a2-35c0b62bd222-serviceca\") pod \"node-ca-h6lp9\" (UID: \"11311f90-d76a-42b5-b6a2-35c0b62bd222\") " pod="openshift-image-registry/node-ca-h6lp9" Oct 30 18:01:41.916435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916340 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31af1374-0d44-4636-bb2a-8e3836ac56b2-data-volume\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:41.916435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916368 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-etc-selinux\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:41.916435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916412 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p496j\" (UniqueName: \"kubernetes.io/projected/76146060-cfc6-4765-82a3-5c5c7601dfda-kube-api-access-p496j\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:41.916435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916430 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/40dd1611-7ec9-411f-920d-e79df7273b90-agent-certs\") pod \"konnectivity-agent-zjlpz\" (UID: \"40dd1611-7ec9-411f-920d-e79df7273b90\") " pod="kube-system/konnectivity-agent-zjlpz" Oct 30 18:01:41.916785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916430 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/31af1374-0d44-4636-bb2a-8e3836ac56b2-data-volume\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:41.916785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916452 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbr88\" (UniqueName: \"kubernetes.io/projected/8d711c2d-0597-4d83-a884-83e50d98378c-kube-api-access-rbr88\") pod \"node-resolver-td6vr\" (UID: \"8d711c2d-0597-4d83-a884-83e50d98378c\") " pod="openshift-dns/node-resolver-td6vr" Oct 30 18:01:41.916785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916472 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-etc-kubernetes\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.916785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916488 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-tls\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:41.916785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916551 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.916785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916587 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31af1374-0d44-4636-bb2a-8e3836ac56b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:41.916785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916614 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-registration-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:41.916785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916639 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:01:41.916785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916663 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-run-netns\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.916785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916688 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-var-lib-kubelet\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.916785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916743 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-tls\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:41.916785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916749 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/31af1374-0d44-4636-bb2a-8e3836ac56b2-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:41.917107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916782 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-run-ovn\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.917107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916820 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-cni-binary-copy\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:01:41.917107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916825 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pkx2s\" (UniqueName: \"kubernetes.io/projected/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-kube-api-access-pkx2s\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:01:41.917107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916941 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d711c2d-0597-4d83-a884-83e50d98378c-tmp-dir\") pod \"node-resolver-td6vr\" (UID: \"8d711c2d-0597-4d83-a884-83e50d98378c\") " pod="openshift-dns/node-resolver-td6vr" Oct 30 18:01:41.917107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.916979 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfmlm\" (UniqueName: \"kubernetes.io/projected/11311f90-d76a-42b5-b6a2-35c0b62bd222-kube-api-access-bfmlm\") pod \"node-ca-h6lp9\" (UID: \"11311f90-d76a-42b5-b6a2-35c0b62bd222\") " pod="openshift-image-registry/node-ca-h6lp9" Oct 30 18:01:41.917107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.917086 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-multus-cni-dir\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.917263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.917105 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d711c2d-0597-4d83-a884-83e50d98378c-tmp-dir\") pod \"node-resolver-td6vr\" (UID: \"8d711c2d-0597-4d83-a884-83e50d98378c\") " pod="openshift-dns/node-resolver-td6vr" Oct 30 18:01:41.917263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.917125 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-hostroot\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.917263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.917143 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-run-systemd\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.917263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.917160 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d458e1b-228c-4e8e-b66f-9688583b39db-env-overrides\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.917263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.917179 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-sysconfig\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:41.917263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.917207 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/17beb356-34fc-4c4a-97a4-72f00c649e7b-root\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:41.917417 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.917313 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/6d458e1b-228c-4e8e-b66f-9688583b39db-env-overrides\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.927282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.927259 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nstb\" (UniqueName: \"kubernetes.io/projected/8754b620-b515-4ee4-9055-469a81126c9e-kube-api-access-2nstb\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:41.927560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.927544 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8clmh\" (UniqueName: \"kubernetes.io/projected/54bdc818-2b49-4f72-a168-6958864242fd-kube-api-access-8clmh\") pod \"dns-default-st6k6\" (UID: \"54bdc818-2b49-4f72-a168-6958864242fd\") " pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:41.928050 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.928034 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsj7n\" (UniqueName: \"kubernetes.io/projected/5e5fca56-5e6b-404d-84b2-4a876f5034cd-kube-api-access-rsj7n\") pod \"iptables-alerter-t7sqv\" (UID: \"5e5fca56-5e6b-404d-84b2-4a876f5034cd\") " pod="openshift-network-operator/iptables-alerter-t7sqv" Oct 30 18:01:41.929441 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.929421 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grpvw\" (UniqueName: \"kubernetes.io/projected/17beb356-34fc-4c4a-97a4-72f00c649e7b-kube-api-access-grpvw\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:41.930865 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.930844 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4xwn\" (UniqueName: \"kubernetes.io/projected/4627766d-20a1-46cd-baf9-cff7cc120679-kube-api-access-q4xwn\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:41.931018 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.930988 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dtfcr\" (UniqueName: \"kubernetes.io/projected/914c8c09-aa3d-4543-9d91-fa443a09b1ad-kube-api-access-dtfcr\") pod \"network-metrics-daemon-g8h9t\" (UID: \"914c8c09-aa3d-4543-9d91-fa443a09b1ad\") " pod="openshift-multus/network-metrics-daemon-g8h9t" Oct 30 18:01:41.931087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.931072 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx4m6\" (UniqueName: \"kubernetes.io/projected/abadb041-5d27-42f8-9cb9-35b78ec5f3bf-kube-api-access-qx4m6\") pod \"metrics-server-866f94cc5d-rv8z2\" (UID: \"abadb041-5d27-42f8-9cb9-35b78ec5f3bf\") " pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:01:41.936814 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.936794 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pm2mh\" (UniqueName: \"kubernetes.io/projected/6d458e1b-228c-4e8e-b66f-9688583b39db-kube-api-access-pm2mh\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:41.936962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.936940 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pkx2s\" (UniqueName: \"kubernetes.io/projected/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-kube-api-access-pkx2s\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:01:41.937201 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.937174 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75w7g\" (UniqueName: \"kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-kube-api-access-75w7g\") pod \"alertmanager-main-0\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:01:41.937287 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.937249 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75xlp\" (UniqueName: \"kubernetes.io/projected/a3c3fe6d-7fab-4cea-a04e-693b985ceb49-kube-api-access-75xlp\") pod \"network-check-target-94r4x\" (UID: \"a3c3fe6d-7fab-4cea-a04e-693b985ceb49\") " pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:01:41.937376 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.937355 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfmlm\" (UniqueName: \"kubernetes.io/projected/11311f90-d76a-42b5-b6a2-35c0b62bd222-kube-api-access-bfmlm\") pod \"node-ca-h6lp9\" (UID: \"11311f90-d76a-42b5-b6a2-35c0b62bd222\") " pod="openshift-image-registry/node-ca-h6lp9" Oct 30 18:01:41.937499 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.937389 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbr88\" (UniqueName: \"kubernetes.io/projected/8d711c2d-0597-4d83-a884-83e50d98378c-kube-api-access-rbr88\") pod \"node-resolver-td6vr\" (UID: \"8d711c2d-0597-4d83-a884-83e50d98378c\") " pod="openshift-dns/node-resolver-td6vr" Oct 30 18:01:41.937499 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.937443 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p496j\" (UniqueName: \"kubernetes.io/projected/76146060-cfc6-4765-82a3-5c5c7601dfda-kube-api-access-p496j\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:41.937499 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.937494 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rjmx\" (UniqueName: \"kubernetes.io/projected/7bd897e6-be5e-4d00-8faf-b2544c55a3f6-kube-api-access-4rjmx\") pod \"ingress-canary-kqvrz\" (UID: \"7bd897e6-be5e-4d00-8faf-b2544c55a3f6\") " pod="openshift-ingress-canary/ingress-canary-kqvrz" Oct 30 18:01:41.937667 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:41.937623 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bhp2k\" (UniqueName: \"kubernetes.io/projected/31af1374-0d44-4636-bb2a-8e3836ac56b2-kube-api-access-bhp2k\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:42.018576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018472 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-kubelet-config\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:42.018576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018502 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-sys-fs\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:42.018576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018552 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-kubelet\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.018576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018571 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-systemd\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.018780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018584 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11311f90-d76a-42b5-b6a2-35c0b62bd222-host\") pod \"node-ca-h6lp9\" (UID: \"11311f90-d76a-42b5-b6a2-35c0b62bd222\") " pod="openshift-image-registry/node-ca-h6lp9" Oct 30 18:01:42.018780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018597 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-kubelet-config\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:42.018780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018604 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-sys-fs\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:42.018780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018601 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-etc-openvswitch\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.018780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018633 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-etc-openvswitch\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.018780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018641 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-systemd\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.018780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018644 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-node-log\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.018780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018666 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/11311f90-d76a-42b5-b6a2-35c0b62bd222-host\") pod \"node-ca-h6lp9\" (UID: \"11311f90-d76a-42b5-b6a2-35c0b62bd222\") " pod="openshift-image-registry/node-ca-h6lp9" Oct 30 18:01:42.018780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018675 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-kubelet\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.018780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018701 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-device-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:42.018780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018712 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-node-log\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.018780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018773 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-device-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:42.018780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018785 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-system-cni-dir\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018804 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-var-lib-cni-bin\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018820 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-cni-bin\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018833 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-systemd-units\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018847 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d711c2d-0597-4d83-a884-83e50d98378c-hosts-file\") pod \"node-resolver-td6vr\" (UID: \"8d711c2d-0597-4d83-a884-83e50d98378c\") " pod="openshift-dns/node-resolver-td6vr" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018848 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-var-lib-cni-bin\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018869 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-cni-bin\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018881 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-system-cni-dir\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018895 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-systemd-units\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018889 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/8d711c2d-0597-4d83-a884-83e50d98378c-hosts-file\") pod \"node-resolver-td6vr\" (UID: \"8d711c2d-0597-4d83-a884-83e50d98378c\") " pod="openshift-dns/node-resolver-td6vr" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018916 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17beb356-34fc-4c4a-97a4-72f00c649e7b-sys\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018899 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/17beb356-34fc-4c4a-97a4-72f00c649e7b-sys\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018938 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-modprobe-d\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018956 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-run\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018970 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.018987 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-dbus\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019001 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-var-lib-openvswitch\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019016 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019032 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-var-lib-kubelet\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019051 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-run\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019057 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-tuning-conf-dir\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019063 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-multus-socket-dir-parent\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019070 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/b0df4b9d-89ce-4bb4-b759-68f1b499b52a-dbus\") pod \"global-pull-secret-syncer-brgpf\" (UID: \"b0df4b9d-89ce-4bb4-b759-68f1b499b52a\") " pod="kube-system/global-pull-secret-syncer-brgpf" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019085 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-var-lib-openvswitch\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019102 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-var-lib-kubelet\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019100 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-run-netns\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019118 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-multus-socket-dir-parent\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019125 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-modprobe-d\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019131 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019135 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31af1374-0d44-4636-bb2a-8e3836ac56b2-crio-socket\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019161 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/31af1374-0d44-4636-bb2a-8e3836ac56b2-crio-socket\") pod \"insights-runtime-extractor-b9kd2\" (UID: \"31af1374-0d44-4636-bb2a-8e3836ac56b2\") " pod="openshift-insights/insights-runtime-extractor-b9kd2" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019129 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-run-netns\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019190 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-etc-selinux\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019233 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-etc-kubernetes\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019235 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-etc-selinux\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:42.019233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019253 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-registration-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019277 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-run-netns\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019273 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-etc-kubernetes\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019300 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-registration-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019306 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-var-lib-kubelet\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019327 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-run-ovn\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019344 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-multus-cni-dir\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019357 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-hostroot\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019374 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-run-systemd\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019384 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-run-ovn\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019394 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-run-netns\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019398 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-sysconfig\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019376 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-var-lib-kubelet\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019414 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-run-systemd\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019420 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/17beb356-34fc-4c4a-97a4-72f00c649e7b-root\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019426 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-hostroot\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019416 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-multus-cni-dir\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019464 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-sysconfig\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019454 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/17beb356-34fc-4c4a-97a4-72f00c649e7b-root\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019533 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-var-lib-cni-multus\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019555 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e5fca56-5e6b-404d-84b2-4a876f5034cd-host-slash\") pod \"iptables-alerter-t7sqv\" (UID: \"5e5fca56-5e6b-404d-84b2-4a876f5034cd\") " pod="openshift-network-operator/iptables-alerter-t7sqv" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019570 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-run-multus-certs\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019585 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-sysctl-conf\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019570 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-var-lib-cni-multus\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019618 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-run-multus-certs\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019630 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/5e5fca56-5e6b-404d-84b2-4a876f5034cd-host-slash\") pod \"iptables-alerter-t7sqv\" (UID: \"5e5fca56-5e6b-404d-84b2-4a876f5034cd\") " pod="openshift-network-operator/iptables-alerter-t7sqv" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019641 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-system-cni-dir\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019659 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-multus-conf-dir\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019673 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-wtmp\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019674 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-sysctl-conf\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019689 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-sysctl-d\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019717 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-multus-conf-dir\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019721 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-system-cni-dir\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019733 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-lib-modules\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019746 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/17beb356-34fc-4c4a-97a4-72f00c649e7b-node-exporter-wtmp\") pod \"node-exporter-5zplp\" (UID: \"17beb356-34fc-4c4a-97a4-72f00c649e7b\") " pod="openshift-monitoring/node-exporter-5zplp" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019785 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-sys\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019751 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-sys\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019824 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-sysctl-d\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019826 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-lib-modules\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019825 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-cnibin\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019853 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc-cnibin\") pod \"multus-additional-cni-plugins-mw76k\" (UID: \"54e2e6ce-5ca5-4de1-b5f6-5a62ad67f3cc\") " pod="openshift-multus/multus-additional-cni-plugins-mw76k" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019865 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-run-openvswitch\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019888 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-cni-netd\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019904 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-slash\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019910 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-run-openvswitch\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019919 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-socket-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019935 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-run-k8s-cni-cncf-io\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019959 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-slash\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019964 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-log-socket\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019972 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-socket-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019980 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-host-run-k8s-cni-cncf-io\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019943 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-cni-netd\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019991 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-log-socket\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.019990 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-cnibin\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.020032 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8754b620-b515-4ee4-9055-469a81126c9e-cnibin\") pod \"multus-9bq58\" (UID: \"8754b620-b515-4ee4-9055-469a81126c9e\") " pod="openshift-multus/multus-9bq58" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.020031 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-run-ovn-kubernetes\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.020061 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6d458e1b-228c-4e8e-b66f-9688583b39db-host-run-ovn-kubernetes\") pod \"ovnkube-node-m556r\" (UID: \"6d458e1b-228c-4e8e-b66f-9688583b39db\") " pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.020062 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.020095 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-kubernetes\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.020103 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4627766d-20a1-46cd-baf9-cff7cc120679-kubelet-dir\") pod \"aws-ebs-csi-driver-node-qdzv5\" (UID: \"4627766d-20a1-46cd-baf9-cff7cc120679\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-qdzv5" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.020136 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-host\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.020140 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-etc-kubernetes\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.020304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.020189 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/76146060-cfc6-4765-82a3-5c5c7601dfda-host\") pod \"tuned-k8684\" (UID: \"76146060-cfc6-4765-82a3-5c5c7601dfda\") " pod="openshift-cluster-node-tuning-operator/tuned-k8684" Oct 30 18:01:42.774631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:42.774597 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-st6k6" Oct 30 18:01:43.261534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:43.261485 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:43.549160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:43.549083 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:01:50.156011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:50.155983 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:01:50.156465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:50.156178 5232 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 18:01:53.640138 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:01:53.640060 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zjlpz" Oct 30 18:01:57.031300 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:01:57.031260 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:02:01.949040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:01.949006 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-866f94cc5d-rv8z2" Oct 30 18:02:06.527732 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:06.527702 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-m556r" Oct 30 18:02:07.236882 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:07.236850 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:02:07.237537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:07.237476 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="prom-label-proxy" containerID="cri-o://2e69cc3749958466d3237c968c19834fbbdd29bd8f51a3283d40ad261254985c" gracePeriod=120 Oct 30 18:02:07.237713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:07.237476 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="alertmanager" containerID="cri-o://7c013b745e3c4229cd540ac7bb7c48da79c793355bea6343f379e52920c80797" gracePeriod=120 Oct 30 18:02:07.237787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:07.237476 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="kube-rbac-proxy-metric" containerID="cri-o://c48d172211a010718301f9fdc8458788649f3859e0b9c223286fec5a4da6f9f4" gracePeriod=120 Oct 30 18:02:07.237844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:07.237525 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="kube-rbac-proxy-web" containerID="cri-o://402e2ba9f11434e02f7191db2e717ad08d830b74f65ecf12e8f5921302c957b6" gracePeriod=120 Oct 30 18:02:07.237887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:07.237552 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="kube-rbac-proxy" containerID="cri-o://e417d8c7b420a656699110cdd3e941d54ee770055feddf0843d95aa95671a3f7" gracePeriod=120 Oct 30 18:02:07.237926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:07.237576 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="config-reloader" containerID="cri-o://ff8ee0a5a16476997d930a3db3141259151ea1391a2762dd3b8f51fcb3845d46" gracePeriod=120 Oct 30 18:02:07.935685 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:07.935655 5232 generic.go:356] "Generic (PLEG): container finished" podID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerID="2e69cc3749958466d3237c968c19834fbbdd29bd8f51a3283d40ad261254985c" exitCode=0 Oct 30 18:02:07.935685 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:07.935681 5232 generic.go:356] "Generic (PLEG): container finished" podID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerID="c48d172211a010718301f9fdc8458788649f3859e0b9c223286fec5a4da6f9f4" exitCode=0 Oct 30 18:02:07.935685 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:07.935689 5232 generic.go:356] "Generic (PLEG): container finished" podID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerID="e417d8c7b420a656699110cdd3e941d54ee770055feddf0843d95aa95671a3f7" exitCode=0 Oct 30 18:02:07.936111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:07.935696 5232 generic.go:356] "Generic (PLEG): container finished" podID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerID="ff8ee0a5a16476997d930a3db3141259151ea1391a2762dd3b8f51fcb3845d46" exitCode=0 Oct 30 18:02:07.936111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:07.935703 5232 generic.go:356] "Generic (PLEG): container finished" podID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerID="7c013b745e3c4229cd540ac7bb7c48da79c793355bea6343f379e52920c80797" exitCode=0 Oct 30 18:02:07.936111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:07.935733 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb82def8-d679-4f36-8d87-b9b23865e3de","Type":"ContainerDied","Data":"2e69cc3749958466d3237c968c19834fbbdd29bd8f51a3283d40ad261254985c"} Oct 30 18:02:07.936111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:07.935820 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb82def8-d679-4f36-8d87-b9b23865e3de","Type":"ContainerDied","Data":"c48d172211a010718301f9fdc8458788649f3859e0b9c223286fec5a4da6f9f4"} Oct 30 18:02:07.936111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:07.935846 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb82def8-d679-4f36-8d87-b9b23865e3de","Type":"ContainerDied","Data":"e417d8c7b420a656699110cdd3e941d54ee770055feddf0843d95aa95671a3f7"} Oct 30 18:02:07.936111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:07.935871 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb82def8-d679-4f36-8d87-b9b23865e3de","Type":"ContainerDied","Data":"ff8ee0a5a16476997d930a3db3141259151ea1391a2762dd3b8f51fcb3845d46"} Oct 30 18:02:07.936111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:07.935896 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb82def8-d679-4f36-8d87-b9b23865e3de","Type":"ContainerDied","Data":"7c013b745e3c4229cd540ac7bb7c48da79c793355bea6343f379e52920c80797"} Oct 30 18:02:08.474936 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.474914 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:08.580111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.580045 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-main-tls\") pod \"fb82def8-d679-4f36-8d87-b9b23865e3de\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " Oct 30 18:02:08.580111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.580081 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-cluster-tls-config\") pod \"fb82def8-d679-4f36-8d87-b9b23865e3de\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " Oct 30 18:02:08.580111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.580106 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-main-db\") pod \"fb82def8-d679-4f36-8d87-b9b23865e3de\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " Oct 30 18:02:08.580429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.580125 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-config-out\") pod \"fb82def8-d679-4f36-8d87-b9b23865e3de\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " Oct 30 18:02:08.580429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.580143 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-web\") pod \"fb82def8-d679-4f36-8d87-b9b23865e3de\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " Oct 30 18:02:08.580429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.580166 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-trusted-ca-bundle\") pod \"fb82def8-d679-4f36-8d87-b9b23865e3de\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " Oct 30 18:02:08.580429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.580197 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-config-volume\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-config-volume\") pod \"fb82def8-d679-4f36-8d87-b9b23865e3de\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " Oct 30 18:02:08.580429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.580224 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-metrics-client-ca\") pod \"fb82def8-d679-4f36-8d87-b9b23865e3de\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " Oct 30 18:02:08.580429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.580261 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-web-config\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-web-config\") pod \"fb82def8-d679-4f36-8d87-b9b23865e3de\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " Oct 30 18:02:08.580429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.580312 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-kube-api-access-75w7g\" (UniqueName: \"kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-kube-api-access-75w7g\") pod \"fb82def8-d679-4f36-8d87-b9b23865e3de\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " Oct 30 18:02:08.580429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.580356 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy\") pod \"fb82def8-d679-4f36-8d87-b9b23865e3de\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " Oct 30 18:02:08.580429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.580386 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-metric\") pod \"fb82def8-d679-4f36-8d87-b9b23865e3de\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " Oct 30 18:02:08.580429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.580413 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-tls-assets\" (UniqueName: \"kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-tls-assets\") pod \"fb82def8-d679-4f36-8d87-b9b23865e3de\" (UID: \"fb82def8-d679-4f36-8d87-b9b23865e3de\") " Oct 30 18:02:08.580916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.580486 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-main-db" pod "fb82def8-d679-4f36-8d87-b9b23865e3de" (UID: "fb82def8-d679-4f36-8d87-b9b23865e3de"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:02:08.580916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.580738 5232 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-main-db\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:02:08.581458 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.581377 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-trusted-ca-bundle" pod "fb82def8-d679-4f36-8d87-b9b23865e3de" (UID: "fb82def8-d679-4f36-8d87-b9b23865e3de"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:08.582042 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.581994 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-metrics-client-ca" pod "fb82def8-d679-4f36-8d87-b9b23865e3de" (UID: "fb82def8-d679-4f36-8d87-b9b23865e3de"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:02:08.584755 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.584709 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-main-tls" pod "fb82def8-d679-4f36-8d87-b9b23865e3de" (UID: "fb82def8-d679-4f36-8d87-b9b23865e3de"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:08.584755 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.584715 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-config-out" pod "fb82def8-d679-4f36-8d87-b9b23865e3de" (UID: "fb82def8-d679-4f36-8d87-b9b23865e3de"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:02:08.585128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.585086 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-tls-assets" pod "fb82def8-d679-4f36-8d87-b9b23865e3de" (UID: "fb82def8-d679-4f36-8d87-b9b23865e3de"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:02:08.585209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.585171 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-metric" pod "fb82def8-d679-4f36-8d87-b9b23865e3de" (UID: "fb82def8-d679-4f36-8d87-b9b23865e3de"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:08.585429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.585349 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-kube-api-access-75w7g" pod "fb82def8-d679-4f36-8d87-b9b23865e3de" (UID: "fb82def8-d679-4f36-8d87-b9b23865e3de"). InnerVolumeSpecName "kube-api-access-75w7g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:02:08.585429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.585406 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy" pod "fb82def8-d679-4f36-8d87-b9b23865e3de" (UID: "fb82def8-d679-4f36-8d87-b9b23865e3de"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:08.585578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.585484 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-config-volume" pod "fb82def8-d679-4f36-8d87-b9b23865e3de" (UID: "fb82def8-d679-4f36-8d87-b9b23865e3de"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:08.585623 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.585586 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-web" pod "fb82def8-d679-4f36-8d87-b9b23865e3de" (UID: "fb82def8-d679-4f36-8d87-b9b23865e3de"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:08.588032 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.588007 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-cluster-tls-config" pod "fb82def8-d679-4f36-8d87-b9b23865e3de" (UID: "fb82def8-d679-4f36-8d87-b9b23865e3de"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:08.595133 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.595110 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-web-config" pod "fb82def8-d679-4f36-8d87-b9b23865e3de" (UID: "fb82def8-d679-4f36-8d87-b9b23865e3de"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:02:08.681033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.681000 5232 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-web-config\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:02:08.681033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.681027 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75w7g\" (UniqueName: \"kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-kube-api-access-75w7g\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:02:08.681033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.681036 5232 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:02:08.681247 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.681046 5232 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:02:08.681247 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.681059 5232 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/fb82def8-d679-4f36-8d87-b9b23865e3de-tls-assets\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:02:08.681247 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.681067 5232 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-main-tls\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:02:08.681247 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.681076 5232 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-cluster-tls-config\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:02:08.681247 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.681084 5232 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/fb82def8-d679-4f36-8d87-b9b23865e3de-config-out\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:02:08.681247 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.681092 5232 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:02:08.681247 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.681101 5232 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:02:08.681247 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.681109 5232 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/fb82def8-d679-4f36-8d87-b9b23865e3de-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:02:08.681247 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.681117 5232 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb82def8-d679-4f36-8d87-b9b23865e3de-metrics-client-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:02:08.942131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.942049 5232 generic.go:356] "Generic (PLEG): container finished" podID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerID="402e2ba9f11434e02f7191db2e717ad08d830b74f65ecf12e8f5921302c957b6" exitCode=0 Oct 30 18:02:08.942554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.942122 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb82def8-d679-4f36-8d87-b9b23865e3de","Type":"ContainerDied","Data":"402e2ba9f11434e02f7191db2e717ad08d830b74f65ecf12e8f5921302c957b6"} Oct 30 18:02:08.942554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.942139 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:08.942554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.942194 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"fb82def8-d679-4f36-8d87-b9b23865e3de","Type":"ContainerDied","Data":"f1fe730a2abe2619857d71bec2084370e75929038ce5a74a4fa4b6d7113aab97"} Oct 30 18:02:08.942554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.942228 5232 scope.go:122] "RemoveContainer" containerID="2e69cc3749958466d3237c968c19834fbbdd29bd8f51a3283d40ad261254985c" Oct 30 18:02:08.949568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.949552 5232 scope.go:122] "RemoveContainer" containerID="c48d172211a010718301f9fdc8458788649f3859e0b9c223286fec5a4da6f9f4" Oct 30 18:02:08.956149 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.956130 5232 scope.go:122] "RemoveContainer" containerID="e417d8c7b420a656699110cdd3e941d54ee770055feddf0843d95aa95671a3f7" Oct 30 18:02:08.962611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.962592 5232 scope.go:122] "RemoveContainer" containerID="402e2ba9f11434e02f7191db2e717ad08d830b74f65ecf12e8f5921302c957b6" Oct 30 18:02:08.968912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.968888 5232 scope.go:122] "RemoveContainer" containerID="ff8ee0a5a16476997d930a3db3141259151ea1391a2762dd3b8f51fcb3845d46" Oct 30 18:02:08.976277 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.976261 5232 scope.go:122] "RemoveContainer" containerID="7c013b745e3c4229cd540ac7bb7c48da79c793355bea6343f379e52920c80797" Oct 30 18:02:08.982820 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.982799 5232 scope.go:122] "RemoveContainer" containerID="eae7e933f2eb2fa8270d189d2f72af5847ed46a7ac5e26e71695065c7a25e4fd" Oct 30 18:02:08.989224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.989204 5232 scope.go:122] "RemoveContainer" containerID="2e69cc3749958466d3237c968c19834fbbdd29bd8f51a3283d40ad261254985c" Oct 30 18:02:08.989515 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:02:08.989483 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e69cc3749958466d3237c968c19834fbbdd29bd8f51a3283d40ad261254985c\": container with ID starting with 2e69cc3749958466d3237c968c19834fbbdd29bd8f51a3283d40ad261254985c not found: ID does not exist" containerID="2e69cc3749958466d3237c968c19834fbbdd29bd8f51a3283d40ad261254985c" Oct 30 18:02:08.989574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.989530 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e69cc3749958466d3237c968c19834fbbdd29bd8f51a3283d40ad261254985c"} err="failed to get container status \"2e69cc3749958466d3237c968c19834fbbdd29bd8f51a3283d40ad261254985c\": rpc error: code = NotFound desc = could not find container \"2e69cc3749958466d3237c968c19834fbbdd29bd8f51a3283d40ad261254985c\": container with ID starting with 2e69cc3749958466d3237c968c19834fbbdd29bd8f51a3283d40ad261254985c not found: ID does not exist" Oct 30 18:02:08.989574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.989564 5232 scope.go:122] "RemoveContainer" containerID="c48d172211a010718301f9fdc8458788649f3859e0b9c223286fec5a4da6f9f4" Oct 30 18:02:08.989797 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:02:08.989781 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c48d172211a010718301f9fdc8458788649f3859e0b9c223286fec5a4da6f9f4\": container with ID starting with c48d172211a010718301f9fdc8458788649f3859e0b9c223286fec5a4da6f9f4 not found: ID does not exist" containerID="c48d172211a010718301f9fdc8458788649f3859e0b9c223286fec5a4da6f9f4" Oct 30 18:02:08.989845 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.989803 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c48d172211a010718301f9fdc8458788649f3859e0b9c223286fec5a4da6f9f4"} err="failed to get container status \"c48d172211a010718301f9fdc8458788649f3859e0b9c223286fec5a4da6f9f4\": rpc error: code = NotFound desc = could not find container \"c48d172211a010718301f9fdc8458788649f3859e0b9c223286fec5a4da6f9f4\": container with ID starting with c48d172211a010718301f9fdc8458788649f3859e0b9c223286fec5a4da6f9f4 not found: ID does not exist" Oct 30 18:02:08.989845 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.989816 5232 scope.go:122] "RemoveContainer" containerID="e417d8c7b420a656699110cdd3e941d54ee770055feddf0843d95aa95671a3f7" Oct 30 18:02:08.990015 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:02:08.989999 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e417d8c7b420a656699110cdd3e941d54ee770055feddf0843d95aa95671a3f7\": container with ID starting with e417d8c7b420a656699110cdd3e941d54ee770055feddf0843d95aa95671a3f7 not found: ID does not exist" containerID="e417d8c7b420a656699110cdd3e941d54ee770055feddf0843d95aa95671a3f7" Oct 30 18:02:08.990054 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.990021 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e417d8c7b420a656699110cdd3e941d54ee770055feddf0843d95aa95671a3f7"} err="failed to get container status \"e417d8c7b420a656699110cdd3e941d54ee770055feddf0843d95aa95671a3f7\": rpc error: code = NotFound desc = could not find container \"e417d8c7b420a656699110cdd3e941d54ee770055feddf0843d95aa95671a3f7\": container with ID starting with e417d8c7b420a656699110cdd3e941d54ee770055feddf0843d95aa95671a3f7 not found: ID does not exist" Oct 30 18:02:08.990054 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.990035 5232 scope.go:122] "RemoveContainer" containerID="402e2ba9f11434e02f7191db2e717ad08d830b74f65ecf12e8f5921302c957b6" Oct 30 18:02:08.990248 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:02:08.990233 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"402e2ba9f11434e02f7191db2e717ad08d830b74f65ecf12e8f5921302c957b6\": container with ID starting with 402e2ba9f11434e02f7191db2e717ad08d830b74f65ecf12e8f5921302c957b6 not found: ID does not exist" containerID="402e2ba9f11434e02f7191db2e717ad08d830b74f65ecf12e8f5921302c957b6" Oct 30 18:02:08.990291 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.990251 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"402e2ba9f11434e02f7191db2e717ad08d830b74f65ecf12e8f5921302c957b6"} err="failed to get container status \"402e2ba9f11434e02f7191db2e717ad08d830b74f65ecf12e8f5921302c957b6\": rpc error: code = NotFound desc = could not find container \"402e2ba9f11434e02f7191db2e717ad08d830b74f65ecf12e8f5921302c957b6\": container with ID starting with 402e2ba9f11434e02f7191db2e717ad08d830b74f65ecf12e8f5921302c957b6 not found: ID does not exist" Oct 30 18:02:08.990291 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.990263 5232 scope.go:122] "RemoveContainer" containerID="ff8ee0a5a16476997d930a3db3141259151ea1391a2762dd3b8f51fcb3845d46" Oct 30 18:02:08.990450 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:02:08.990435 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff8ee0a5a16476997d930a3db3141259151ea1391a2762dd3b8f51fcb3845d46\": container with ID starting with ff8ee0a5a16476997d930a3db3141259151ea1391a2762dd3b8f51fcb3845d46 not found: ID does not exist" containerID="ff8ee0a5a16476997d930a3db3141259151ea1391a2762dd3b8f51fcb3845d46" Oct 30 18:02:08.990485 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.990454 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff8ee0a5a16476997d930a3db3141259151ea1391a2762dd3b8f51fcb3845d46"} err="failed to get container status \"ff8ee0a5a16476997d930a3db3141259151ea1391a2762dd3b8f51fcb3845d46\": rpc error: code = NotFound desc = could not find container \"ff8ee0a5a16476997d930a3db3141259151ea1391a2762dd3b8f51fcb3845d46\": container with ID starting with ff8ee0a5a16476997d930a3db3141259151ea1391a2762dd3b8f51fcb3845d46 not found: ID does not exist" Oct 30 18:02:08.990485 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.990467 5232 scope.go:122] "RemoveContainer" containerID="7c013b745e3c4229cd540ac7bb7c48da79c793355bea6343f379e52920c80797" Oct 30 18:02:08.990912 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:02:08.990893 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c013b745e3c4229cd540ac7bb7c48da79c793355bea6343f379e52920c80797\": container with ID starting with 7c013b745e3c4229cd540ac7bb7c48da79c793355bea6343f379e52920c80797 not found: ID does not exist" containerID="7c013b745e3c4229cd540ac7bb7c48da79c793355bea6343f379e52920c80797" Oct 30 18:02:08.990992 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.990914 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c013b745e3c4229cd540ac7bb7c48da79c793355bea6343f379e52920c80797"} err="failed to get container status \"7c013b745e3c4229cd540ac7bb7c48da79c793355bea6343f379e52920c80797\": rpc error: code = NotFound desc = could not find container \"7c013b745e3c4229cd540ac7bb7c48da79c793355bea6343f379e52920c80797\": container with ID starting with 7c013b745e3c4229cd540ac7bb7c48da79c793355bea6343f379e52920c80797 not found: ID does not exist" Oct 30 18:02:08.990992 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.990925 5232 scope.go:122] "RemoveContainer" containerID="eae7e933f2eb2fa8270d189d2f72af5847ed46a7ac5e26e71695065c7a25e4fd" Oct 30 18:02:08.991118 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:02:08.991102 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eae7e933f2eb2fa8270d189d2f72af5847ed46a7ac5e26e71695065c7a25e4fd\": container with ID starting with eae7e933f2eb2fa8270d189d2f72af5847ed46a7ac5e26e71695065c7a25e4fd not found: ID does not exist" containerID="eae7e933f2eb2fa8270d189d2f72af5847ed46a7ac5e26e71695065c7a25e4fd" Oct 30 18:02:08.991152 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.991123 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eae7e933f2eb2fa8270d189d2f72af5847ed46a7ac5e26e71695065c7a25e4fd"} err="failed to get container status \"eae7e933f2eb2fa8270d189d2f72af5847ed46a7ac5e26e71695065c7a25e4fd\": rpc error: code = NotFound desc = could not find container \"eae7e933f2eb2fa8270d189d2f72af5847ed46a7ac5e26e71695065c7a25e4fd\": container with ID starting with eae7e933f2eb2fa8270d189d2f72af5847ed46a7ac5e26e71695065c7a25e4fd not found: ID does not exist" Oct 30 18:02:08.996278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:08.996255 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:02:09.008186 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.008164 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:02:09.032194 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032164 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:02:09.032386 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032374 5232 cpu_manager.go:414] "containerMap: removing container" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="kube-rbac-proxy-web" Oct 30 18:02:09.032435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032387 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="kube-rbac-proxy-web" Oct 30 18:02:09.032435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032395 5232 cpu_manager.go:414] "containerMap: removing container" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="config-reloader" Oct 30 18:02:09.032435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032400 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="config-reloader" Oct 30 18:02:09.032435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032406 5232 cpu_manager.go:414] "containerMap: removing container" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="alertmanager" Oct 30 18:02:09.032435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032411 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="alertmanager" Oct 30 18:02:09.032435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032419 5232 cpu_manager.go:414] "containerMap: removing container" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="prom-label-proxy" Oct 30 18:02:09.032435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032423 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="prom-label-proxy" Oct 30 18:02:09.032435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032431 5232 cpu_manager.go:414] "containerMap: removing container" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="kube-rbac-proxy" Oct 30 18:02:09.032435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032435 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="kube-rbac-proxy" Oct 30 18:02:09.032706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032444 5232 cpu_manager.go:414] "containerMap: removing container" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="kube-rbac-proxy-metric" Oct 30 18:02:09.032706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032449 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="kube-rbac-proxy-metric" Oct 30 18:02:09.032706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032457 5232 cpu_manager.go:414] "containerMap: removing container" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="init-config-reloader" Oct 30 18:02:09.032706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032461 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="init-config-reloader" Oct 30 18:02:09.032706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032499 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="kube-rbac-proxy-web" Oct 30 18:02:09.032706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032525 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="config-reloader" Oct 30 18:02:09.032706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032530 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="alertmanager" Oct 30 18:02:09.032706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032536 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="prom-label-proxy" Oct 30 18:02:09.032706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032541 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="kube-rbac-proxy" Oct 30 18:02:09.032706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032547 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="kube-rbac-proxy-metric" Oct 30 18:02:09.032706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.032553 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" containerName="init-config-reloader" Oct 30 18:02:09.036074 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.036056 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.063415 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.063389 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 18:02:09.063612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.063457 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 18:02:09.063612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.063535 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 18:02:09.063612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.063566 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 18:02:09.063612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.063592 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 18:02:09.063812 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.063614 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-fqgf4\"" Oct 30 18:02:09.063812 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.063623 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 18:02:09.063812 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.063639 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 18:02:09.063812 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.063720 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 18:02:09.063812 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.063723 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 18:02:09.064612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.064589 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:02:09.184948 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.184896 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-config-out\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.184948 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.184955 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.185193 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.185029 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.185193 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.185055 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-config-volume\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.185193 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.185079 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.185193 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.185129 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.185193 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.185155 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-web-config\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.185193 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.185171 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.185422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.185237 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.185422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.185269 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.185422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.185300 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.185422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.185316 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dc9nf\" (UniqueName: \"kubernetes.io/projected/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-kube-api-access-dc9nf\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.185422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.185338 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.286542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.286479 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.286695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.286558 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-config-volume\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.286695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.286586 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.286695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.286610 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.286695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.286640 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-web-config\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.286695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.286668 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.286695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.286692 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.286963 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.286715 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.286963 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.286760 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.286963 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.286785 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dc9nf\" (UniqueName: \"kubernetes.io/projected/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-kube-api-access-dc9nf\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.286963 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.286820 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.286963 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.286852 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-config-out\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.286963 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.286879 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.287261 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.286985 5232 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 18:02:09.287325 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.287254 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.288145 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.288116 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.288299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.288194 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.289936 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.289907 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-config-out\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.290039 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.289968 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.290039 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.289979 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.290213 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.290192 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.290357 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.290339 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-tls-assets\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.290636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.290616 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-web-config\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.290740 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.290719 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.290841 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.290826 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.291916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.291895 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-config-volume\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.308422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.308394 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dc9nf\" (UniqueName: \"kubernetes.io/projected/54a5c756-d540-4a43-b8a0-aacaf05dc8e8-kube-api-access-dc9nf\") pod \"alertmanager-main-0\" (UID: \"54a5c756-d540-4a43-b8a0-aacaf05dc8e8\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.354332 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.354300 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 18:02:09.669052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.668924 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 18:02:09.671742 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:02:09.671699 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54a5c756_d540_4a43_b8a0_aacaf05dc8e8.slice/crio-562fcbd86a0c9b54d74cad8c2f92d774212b2ba1aed6a43ff6309f0b761a2857 WatchSource:0}: Error finding container 562fcbd86a0c9b54d74cad8c2f92d774212b2ba1aed6a43ff6309f0b761a2857: Status 404 returned error can't find the container with id 562fcbd86a0c9b54d74cad8c2f92d774212b2ba1aed6a43ff6309f0b761a2857 Oct 30 18:02:09.947794 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.947711 5232 generic.go:356] "Generic (PLEG): container finished" podID="54a5c756-d540-4a43-b8a0-aacaf05dc8e8" containerID="e9920dff2e84168cda54a9ac529b2abc025d5371034a5990e4631eb08966da17" exitCode=0 Oct 30 18:02:09.947794 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.947753 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54a5c756-d540-4a43-b8a0-aacaf05dc8e8","Type":"ContainerDied","Data":"e9920dff2e84168cda54a9ac529b2abc025d5371034a5990e4631eb08966da17"} Oct 30 18:02:09.948196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:09.947809 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54a5c756-d540-4a43-b8a0-aacaf05dc8e8","Type":"ContainerStarted","Data":"562fcbd86a0c9b54d74cad8c2f92d774212b2ba1aed6a43ff6309f0b761a2857"} Oct 30 18:02:10.802017 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:10.801975 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb82def8-d679-4f36-8d87-b9b23865e3de" path="/var/lib/kubelet/pods/fb82def8-d679-4f36-8d87-b9b23865e3de/volumes" Oct 30 18:02:10.953577 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:10.953539 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54a5c756-d540-4a43-b8a0-aacaf05dc8e8","Type":"ContainerStarted","Data":"9ede0c92943f197f66d832761ab1c2009519a68c4dc5998c478be3a8d2f231ba"} Oct 30 18:02:10.953946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:10.953604 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54a5c756-d540-4a43-b8a0-aacaf05dc8e8","Type":"ContainerStarted","Data":"daca3b559b15ab093938a1222607e7cd650ac6ff56abaedf30745bd2a05c69ad"} Oct 30 18:02:10.953946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:10.953628 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54a5c756-d540-4a43-b8a0-aacaf05dc8e8","Type":"ContainerStarted","Data":"1c2a4e7d177838bcf44e5a4544834b09e9fde286a558e1267ed9c961bfe5fb1e"} Oct 30 18:02:10.953946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:10.953653 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54a5c756-d540-4a43-b8a0-aacaf05dc8e8","Type":"ContainerStarted","Data":"87d7fb13c1c2b4ed12717fc630f350865a2e95974bccb2c061b68f32f54e2080"} Oct 30 18:02:10.953946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:10.953677 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54a5c756-d540-4a43-b8a0-aacaf05dc8e8","Type":"ContainerStarted","Data":"6d303899fcd8032f6d9d0cfab4f65fc183fd83f18931ee29d73df01e4194d014"} Oct 30 18:02:10.953946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:10.953699 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"54a5c756-d540-4a43-b8a0-aacaf05dc8e8","Type":"ContainerStarted","Data":"a1b9ea7a18d96aca45ffa79ead24b08cb6ba34b065aee310201d5807f3f3b74d"} Oct 30 18:02:11.036570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.036501 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.024438457 podStartE2EDuration="2.036487529s" totalImagesPullingTime="0s" totalInitContainerRuntime="12.049072ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:02:11.036189646 +0000 UTC m=+30.352609967" watchObservedRunningTime="2026-10-30 18:02:11.036487529 +0000 UTC m=+30.352907875" Oct 30 18:02:11.151221 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.151148 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-7d4f86898f-ng4c7"] Oct 30 18:02:11.163300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.163275 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.175991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.175962 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7d4f86898f-ng4c7"] Oct 30 18:02:11.179945 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.179921 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 18:02:11.179945 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.179949 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 18:02:11.180800 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.180778 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 18:02:11.180910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.180861 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-57fn6\"" Oct 30 18:02:11.180955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.180924 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 18:02:11.275481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.275452 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 18:02:11.287958 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.287930 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 18:02:11.305162 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.305139 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-federate-client-tls\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.305243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.305172 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-25sbs\" (UniqueName: \"kubernetes.io/projected/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-kube-api-access-25sbs\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.305243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.305195 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-telemeter-client-tls\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.305345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.305272 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.305345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.305320 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-serving-certs-ca-bundle\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.305451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.305353 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.305451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.305396 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-secret-telemeter-client\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.305574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.305459 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-metrics-client-ca\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.406430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.406350 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-25sbs\" (UniqueName: \"kubernetes.io/projected/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-kube-api-access-25sbs\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.406430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.406390 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-telemeter-client-tls\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.406430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.406409 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.406430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.406429 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-serving-certs-ca-bundle\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.406773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.406543 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.406773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.406602 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-secret-telemeter-client\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.406773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.406650 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-metrics-client-ca\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.406773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.406707 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-federate-client-tls\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.407263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.407222 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-serving-certs-ca-bundle\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.407530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.407387 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-metrics-client-ca\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.407683 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.407619 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-telemeter-trusted-ca-bundle\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.409377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.409349 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-telemeter-client-tls\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.409479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.409391 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.409479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.409420 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-secret-telemeter-client\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.409479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.409438 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-federate-client-tls\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.425686 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.425660 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-25sbs\" (UniqueName: \"kubernetes.io/projected/e8a21a50-4101-4fe4-ba87-ad4aa89f88e7-kube-api-access-25sbs\") pod \"telemeter-client-7d4f86898f-ng4c7\" (UID: \"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7\") " pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.498065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.498034 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" Oct 30 18:02:11.820386 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.820353 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-7d4f86898f-ng4c7"] Oct 30 18:02:11.833244 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:02:11.833216 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode8a21a50_4101_4fe4_ba87_ad4aa89f88e7.slice/crio-20278f21d5889a0dee19b527865df9916dce65952fc927f93dfd4e357fa221ae WatchSource:0}: Error finding container 20278f21d5889a0dee19b527865df9916dce65952fc927f93dfd4e357fa221ae: Status 404 returned error can't find the container with id 20278f21d5889a0dee19b527865df9916dce65952fc927f93dfd4e357fa221ae Oct 30 18:02:11.835317 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.835299 5232 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:02:11.960301 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:11.960266 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" event={"ID":"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7","Type":"ContainerStarted","Data":"20278f21d5889a0dee19b527865df9916dce65952fc927f93dfd4e357fa221ae"} Oct 30 18:02:13.551903 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:13.551874 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-94r4x" Oct 30 18:02:14.973207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:14.973169 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" event={"ID":"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7","Type":"ContainerStarted","Data":"e435b4e54056b7717f211e9c099132bed7bbc27656d051c924ba9bc3a1757c3e"} Oct 30 18:02:14.973747 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:14.973229 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" event={"ID":"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7","Type":"ContainerStarted","Data":"87a5d3c3f36bc97a8bf62e9c18a8a9ab0ea3ef8bb01c1616940996fe29764c7a"} Oct 30 18:02:14.973747 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:14.973250 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" event={"ID":"e8a21a50-4101-4fe4-ba87-ad4aa89f88e7","Type":"ContainerStarted","Data":"7ff58e34a01ea5bb72428a934c7e32a496718552de5644122ae2e5e5bf8aed2f"} Oct 30 18:02:15.040574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:02:15.040522 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-7d4f86898f-ng4c7" podStartSLOduration=1.900213181 podStartE2EDuration="4.040493438s" totalImagesPullingTime="2.140280257s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:02:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:02:15.028269595 +0000 UTC m=+34.344689915" watchObservedRunningTime="2026-10-30 18:02:15.040493438 +0000 UTC m=+34.356913756" Oct 30 18:02:27.031911 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:02:27.031864 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:02:57.031479 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:02:57.031437 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:03:27.032394 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:03:27.032317 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:03:57.032319 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:03:57.032281 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:04:21.507586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.507547 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z"] Oct 30 18:04:21.509737 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.509711 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z" Oct 30 18:04:21.536204 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.536176 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 18:04:21.536362 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.536213 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 18:04:21.536692 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.536672 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z"] Oct 30 18:04:21.537996 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.537982 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"e644421527c169d876fdb59235c927986d6bd2-9d2815fc-dockercfg-s8c27\"" Oct 30 18:04:21.587688 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.587645 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/190b0091-724c-49b6-8fba-5d51315d5736-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z\" (UID: \"190b0091-724c-49b6-8fba-5d51315d5736\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z" Oct 30 18:04:21.587864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.587709 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/190b0091-724c-49b6-8fba-5d51315d5736-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z\" (UID: \"190b0091-724c-49b6-8fba-5d51315d5736\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z" Oct 30 18:04:21.587864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.587791 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5jct\" (UniqueName: \"kubernetes.io/projected/190b0091-724c-49b6-8fba-5d51315d5736-kube-api-access-v5jct\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z\" (UID: \"190b0091-724c-49b6-8fba-5d51315d5736\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z" Oct 30 18:04:21.686772 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.686737 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg"] Oct 30 18:04:21.688646 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.688629 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.688800 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.688762 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/190b0091-724c-49b6-8fba-5d51315d5736-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z\" (UID: \"190b0091-724c-49b6-8fba-5d51315d5736\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z" Oct 30 18:04:21.688976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.688815 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5jct\" (UniqueName: \"kubernetes.io/projected/190b0091-724c-49b6-8fba-5d51315d5736-kube-api-access-v5jct\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z\" (UID: \"190b0091-724c-49b6-8fba-5d51315d5736\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z" Oct 30 18:04:21.688976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.688880 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/190b0091-724c-49b6-8fba-5d51315d5736-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z\" (UID: \"190b0091-724c-49b6-8fba-5d51315d5736\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z" Oct 30 18:04:21.689225 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.689207 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/190b0091-724c-49b6-8fba-5d51315d5736-util\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z\" (UID: \"190b0091-724c-49b6-8fba-5d51315d5736\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z" Oct 30 18:04:21.689316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.689294 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/190b0091-724c-49b6-8fba-5d51315d5736-bundle\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z\" (UID: \"190b0091-724c-49b6-8fba-5d51315d5736\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z" Oct 30 18:04:21.706472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.706446 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 18:04:21.706755 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.706739 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 18:04:21.706825 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.706787 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 18:04:21.706825 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.706788 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 18:04:21.706825 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.706806 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 18:04:21.706936 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.706823 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 18:04:21.707012 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.706999 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-wzxwr\"" Oct 30 18:04:21.710697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.710675 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg"] Oct 30 18:04:21.713554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.713528 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5jct\" (UniqueName: \"kubernetes.io/projected/190b0091-724c-49b6-8fba-5d51315d5736-kube-api-access-v5jct\") pod \"e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z\" (UID: \"190b0091-724c-49b6-8fba-5d51315d5736\") " pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z" Oct 30 18:04:21.789314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.789230 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/09bbbf57-61a9-4403-a805-b86b03119b97-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.789314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.789271 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.789570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.789358 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bbbf57-61a9-4403-a805-b86b03119b97-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.789570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.789386 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.789570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.789405 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r74fk\" (UniqueName: \"kubernetes.io/projected/09bbbf57-61a9-4403-a805-b86b03119b97-kube-api-access-r74fk\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.789570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.789467 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bbbf57-61a9-4403-a805-b86b03119b97-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.789570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.789487 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.822650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.822620 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z" Oct 30 18:04:21.890282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.890240 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.890597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.890574 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r74fk\" (UniqueName: \"kubernetes.io/projected/09bbbf57-61a9-4403-a805-b86b03119b97-kube-api-access-r74fk\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.890755 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.890738 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bbbf57-61a9-4403-a805-b86b03119b97-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.891017 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.890995 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.891097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.891066 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/09bbbf57-61a9-4403-a805-b86b03119b97-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.891153 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.891103 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.891542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.891499 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bbbf57-61a9-4403-a805-b86b03119b97-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.892260 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.892233 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/09bbbf57-61a9-4403-a805-b86b03119b97-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.894032 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.893989 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.894722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.894701 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bbbf57-61a9-4403-a805-b86b03119b97-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.895315 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.895294 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.895785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.895743 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.909967 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.909941 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r74fk\" (UniqueName: \"kubernetes.io/projected/09bbbf57-61a9-4403-a805-b86b03119b97-kube-api-access-r74fk\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.910731 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.910713 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bbbf57-61a9-4403-a805-b86b03119b97-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-7p9tg\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:21.998884 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:21.998854 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:22.111172 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:22.111091 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z"] Oct 30 18:04:22.113775 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:04:22.113738 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod190b0091_724c_49b6_8fba_5d51315d5736.slice/crio-fbba458ab6203e13cfc4fc2c2a0278aee9e70ac58739cdc80a94724c205192aa WatchSource:0}: Error finding container fbba458ab6203e13cfc4fc2c2a0278aee9e70ac58739cdc80a94724c205192aa: Status 404 returned error can't find the container with id fbba458ab6203e13cfc4fc2c2a0278aee9e70ac58739cdc80a94724c205192aa Oct 30 18:04:22.289954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:22.289930 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg"] Oct 30 18:04:22.292072 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:04:22.292044 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09bbbf57_61a9_4403_a805_b86b03119b97.slice/crio-78a9dc7b24f11fc62dbe0f044abcdfde639767ed0d3122bedb70d315252278d6 WatchSource:0}: Error finding container 78a9dc7b24f11fc62dbe0f044abcdfde639767ed0d3122bedb70d315252278d6: Status 404 returned error can't find the container with id 78a9dc7b24f11fc62dbe0f044abcdfde639767ed0d3122bedb70d315252278d6 Oct 30 18:04:22.366430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:22.366395 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" event={"ID":"09bbbf57-61a9-4403-a805-b86b03119b97","Type":"ContainerStarted","Data":"78a9dc7b24f11fc62dbe0f044abcdfde639767ed0d3122bedb70d315252278d6"} Oct 30 18:04:22.367561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:22.367538 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z" event={"ID":"190b0091-724c-49b6-8fba-5d51315d5736","Type":"ContainerStarted","Data":"fbba458ab6203e13cfc4fc2c2a0278aee9e70ac58739cdc80a94724c205192aa"} Oct 30 18:04:25.773041 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:25.773001 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:04:25.773420 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:25.773077 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:04:26.385471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:26.385427 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" event={"ID":"09bbbf57-61a9-4403-a805-b86b03119b97","Type":"ContainerStarted","Data":"dce7618de54b80b24e4a8f8df4c521c5a8ae4fcd8faf89a9b81394ea8b8b765c"} Oct 30 18:04:26.385814 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:26.385786 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:26.387947 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:26.387914 5232 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-7p9tg container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:04:26.388064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:26.387983 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" podUID="09bbbf57-61a9-4403-a805-b86b03119b97" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:04:26.454524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:26.454443 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" podStartSLOduration=1.976330642 podStartE2EDuration="5.45442659s" totalImagesPullingTime="3.478095948s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:04:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:04:26.442368371 +0000 UTC m=+165.758788692" watchObservedRunningTime="2026-10-30 18:04:26.45442659 +0000 UTC m=+165.770846912" Oct 30 18:04:27.036397 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:04:27.036360 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:04:27.390725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:27.390651 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:04:28.395586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:28.395477 5232 generic.go:356] "Generic (PLEG): container finished" podID="190b0091-724c-49b6-8fba-5d51315d5736" containerID="2aecd5e617c8f63965eb9d499f3444708bad897175defb2426c9b50d93acf604" exitCode=0 Oct 30 18:04:28.396096 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:28.395578 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z" event={"ID":"190b0091-724c-49b6-8fba-5d51315d5736","Type":"ContainerDied","Data":"2aecd5e617c8f63965eb9d499f3444708bad897175defb2426c9b50d93acf604"} Oct 30 18:04:29.400862 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:29.400820 5232 generic.go:356] "Generic (PLEG): container finished" podID="190b0091-724c-49b6-8fba-5d51315d5736" containerID="615519c21a4937ef71df9bd5549f1d81e7c7300648cfe03c238303cdbce6b750" exitCode=0 Oct 30 18:04:29.401270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:29.400873 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z" event={"ID":"190b0091-724c-49b6-8fba-5d51315d5736","Type":"ContainerDied","Data":"615519c21a4937ef71df9bd5549f1d81e7c7300648cfe03c238303cdbce6b750"} Oct 30 18:04:35.423382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:35.423296 5232 generic.go:356] "Generic (PLEG): container finished" podID="190b0091-724c-49b6-8fba-5d51315d5736" containerID="0ebf257e2f3c385a037d6c364ecffb0b1c2154dee9e16fdf7e9191bb1d4d32bd" exitCode=0 Oct 30 18:04:35.423382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:35.423367 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z" event={"ID":"190b0091-724c-49b6-8fba-5d51315d5736","Type":"ContainerDied","Data":"0ebf257e2f3c385a037d6c364ecffb0b1c2154dee9e16fdf7e9191bb1d4d32bd"} Oct 30 18:04:36.551279 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:36.551253 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z" Oct 30 18:04:36.618046 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:36.618011 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190b0091-724c-49b6-8fba-5d51315d5736-bundle\" (UniqueName: \"kubernetes.io/empty-dir/190b0091-724c-49b6-8fba-5d51315d5736-bundle\") pod \"190b0091-724c-49b6-8fba-5d51315d5736\" (UID: \"190b0091-724c-49b6-8fba-5d51315d5736\") " Oct 30 18:04:36.618212 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:36.618056 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/190b0091-724c-49b6-8fba-5d51315d5736-kube-api-access-v5jct\" (UniqueName: \"kubernetes.io/projected/190b0091-724c-49b6-8fba-5d51315d5736-kube-api-access-v5jct\") pod \"190b0091-724c-49b6-8fba-5d51315d5736\" (UID: \"190b0091-724c-49b6-8fba-5d51315d5736\") " Oct 30 18:04:36.618212 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:36.618175 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/190b0091-724c-49b6-8fba-5d51315d5736-util\" (UniqueName: \"kubernetes.io/empty-dir/190b0091-724c-49b6-8fba-5d51315d5736-util\") pod \"190b0091-724c-49b6-8fba-5d51315d5736\" (UID: \"190b0091-724c-49b6-8fba-5d51315d5736\") " Oct 30 18:04:36.618664 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:36.618641 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190b0091-724c-49b6-8fba-5d51315d5736-bundle" pod "190b0091-724c-49b6-8fba-5d51315d5736" (UID: "190b0091-724c-49b6-8fba-5d51315d5736"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:36.620498 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:36.620467 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/190b0091-724c-49b6-8fba-5d51315d5736-kube-api-access-v5jct" pod "190b0091-724c-49b6-8fba-5d51315d5736" (UID: "190b0091-724c-49b6-8fba-5d51315d5736"). InnerVolumeSpecName "kube-api-access-v5jct". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:04:36.622150 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:36.622130 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/190b0091-724c-49b6-8fba-5d51315d5736-util" pod "190b0091-724c-49b6-8fba-5d51315d5736" (UID: "190b0091-724c-49b6-8fba-5d51315d5736"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:04:36.719784 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:36.719701 5232 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/190b0091-724c-49b6-8fba-5d51315d5736-util\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:04:36.719784 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:36.719741 5232 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/190b0091-724c-49b6-8fba-5d51315d5736-bundle\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:04:36.719784 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:36.719751 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5jct\" (UniqueName: \"kubernetes.io/projected/190b0091-724c-49b6-8fba-5d51315d5736-kube-api-access-v5jct\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:04:37.431427 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:37.431391 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z" event={"ID":"190b0091-724c-49b6-8fba-5d51315d5736","Type":"ContainerDied","Data":"fbba458ab6203e13cfc4fc2c2a0278aee9e70ac58739cdc80a94724c205192aa"} Oct 30 18:04:37.431427 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:37.431430 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/e644421527c169d876fdb59235c927986d6bd2881f72d188068392be71mml6z" Oct 30 18:04:37.431664 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:04:37.431445 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fbba458ab6203e13cfc4fc2c2a0278aee9e70ac58739cdc80a94724c205192aa" Oct 30 18:04:57.031947 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:04:57.031864 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:05:20.622494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.622461 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd"] Oct 30 18:05:20.622910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.622747 5232 cpu_manager.go:414] "containerMap: removing container" podUID="190b0091-724c-49b6-8fba-5d51315d5736" containerName="pull" Oct 30 18:05:20.622910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.622758 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190b0091-724c-49b6-8fba-5d51315d5736" containerName="pull" Oct 30 18:05:20.622910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.622768 5232 cpu_manager.go:414] "containerMap: removing container" podUID="190b0091-724c-49b6-8fba-5d51315d5736" containerName="extract" Oct 30 18:05:20.622910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.622774 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190b0091-724c-49b6-8fba-5d51315d5736" containerName="extract" Oct 30 18:05:20.622910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.622782 5232 cpu_manager.go:414] "containerMap: removing container" podUID="190b0091-724c-49b6-8fba-5d51315d5736" containerName="util" Oct 30 18:05:20.622910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.622787 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="190b0091-724c-49b6-8fba-5d51315d5736" containerName="util" Oct 30 18:05:20.622910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.622835 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="190b0091-724c-49b6-8fba-5d51315d5736" containerName="extract" Oct 30 18:05:20.624732 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.624717 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.645876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.645842 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd"] Oct 30 18:05:20.681447 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.681416 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 18:05:20.681628 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.681527 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 18:05:20.681899 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.681887 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 18:05:20.682020 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.682001 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 18:05:20.682084 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.682045 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 18:05:20.682218 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.682206 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-hwrgm\"" Oct 30 18:05:20.742038 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.742003 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22f202a4-66e8-4175-bdab-075f0c0667cb-tmp\") pod \"opendatahub-operator-controller-manager-85c77bd65b-kt4qd\" (UID: \"22f202a4-66e8-4175-bdab-075f0c0667cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.742038 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.742049 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22f202a4-66e8-4175-bdab-075f0c0667cb-webhook-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-kt4qd\" (UID: \"22f202a4-66e8-4175-bdab-075f0c0667cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.742244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.742131 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22f202a4-66e8-4175-bdab-075f0c0667cb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-kt4qd\" (UID: \"22f202a4-66e8-4175-bdab-075f0c0667cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.742244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.742231 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/22f202a4-66e8-4175-bdab-075f0c0667cb-metrics-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-kt4qd\" (UID: \"22f202a4-66e8-4175-bdab-075f0c0667cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.742313 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.742260 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-svc5j\" (UniqueName: \"kubernetes.io/projected/22f202a4-66e8-4175-bdab-075f0c0667cb-kube-api-access-svc5j\") pod \"opendatahub-operator-controller-manager-85c77bd65b-kt4qd\" (UID: \"22f202a4-66e8-4175-bdab-075f0c0667cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.742313 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.742286 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22f202a4-66e8-4175-bdab-075f0c0667cb-manifests\") pod \"opendatahub-operator-controller-manager-85c77bd65b-kt4qd\" (UID: \"22f202a4-66e8-4175-bdab-075f0c0667cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.842732 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.842697 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22f202a4-66e8-4175-bdab-075f0c0667cb-webhook-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-kt4qd\" (UID: \"22f202a4-66e8-4175-bdab-075f0c0667cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.842732 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.842734 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22f202a4-66e8-4175-bdab-075f0c0667cb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-kt4qd\" (UID: \"22f202a4-66e8-4175-bdab-075f0c0667cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.842984 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.842762 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/22f202a4-66e8-4175-bdab-075f0c0667cb-metrics-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-kt4qd\" (UID: \"22f202a4-66e8-4175-bdab-075f0c0667cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.842984 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.842778 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-svc5j\" (UniqueName: \"kubernetes.io/projected/22f202a4-66e8-4175-bdab-075f0c0667cb-kube-api-access-svc5j\") pod \"opendatahub-operator-controller-manager-85c77bd65b-kt4qd\" (UID: \"22f202a4-66e8-4175-bdab-075f0c0667cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.842984 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.842797 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22f202a4-66e8-4175-bdab-075f0c0667cb-manifests\") pod \"opendatahub-operator-controller-manager-85c77bd65b-kt4qd\" (UID: \"22f202a4-66e8-4175-bdab-075f0c0667cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.842984 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.842843 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22f202a4-66e8-4175-bdab-075f0c0667cb-tmp\") pod \"opendatahub-operator-controller-manager-85c77bd65b-kt4qd\" (UID: \"22f202a4-66e8-4175-bdab-075f0c0667cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.843306 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.843279 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/22f202a4-66e8-4175-bdab-075f0c0667cb-manifests\") pod \"opendatahub-operator-controller-manager-85c77bd65b-kt4qd\" (UID: \"22f202a4-66e8-4175-bdab-075f0c0667cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.845150 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.845122 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/22f202a4-66e8-4175-bdab-075f0c0667cb-tmp\") pod \"opendatahub-operator-controller-manager-85c77bd65b-kt4qd\" (UID: \"22f202a4-66e8-4175-bdab-075f0c0667cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.845373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.845355 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/22f202a4-66e8-4175-bdab-075f0c0667cb-webhook-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-kt4qd\" (UID: \"22f202a4-66e8-4175-bdab-075f0c0667cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.845568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.845547 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/22f202a4-66e8-4175-bdab-075f0c0667cb-metrics-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-kt4qd\" (UID: \"22f202a4-66e8-4175-bdab-075f0c0667cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.845639 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.845554 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/22f202a4-66e8-4175-bdab-075f0c0667cb-apiservice-cert\") pod \"opendatahub-operator-controller-manager-85c77bd65b-kt4qd\" (UID: \"22f202a4-66e8-4175-bdab-075f0c0667cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.865135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.865111 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-svc5j\" (UniqueName: \"kubernetes.io/projected/22f202a4-66e8-4175-bdab-075f0c0667cb-kube-api-access-svc5j\") pod \"opendatahub-operator-controller-manager-85c77bd65b-kt4qd\" (UID: \"22f202a4-66e8-4175-bdab-075f0c0667cb\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:20.936805 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:20.936725 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:21.283390 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:21.283359 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd"] Oct 30 18:05:21.286844 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:05:21.286815 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22f202a4_66e8_4175_bdab_075f0c0667cb.slice/crio-0bfb8656d10a507234782ef8cea18e4ad468de9882e45f2fbe284841b9a99e2e WatchSource:0}: Error finding container 0bfb8656d10a507234782ef8cea18e4ad468de9882e45f2fbe284841b9a99e2e: Status 404 returned error can't find the container with id 0bfb8656d10a507234782ef8cea18e4ad468de9882e45f2fbe284841b9a99e2e Oct 30 18:05:21.577771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:21.577684 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" event={"ID":"22f202a4-66e8-4175-bdab-075f0c0667cb","Type":"ContainerStarted","Data":"0bfb8656d10a507234782ef8cea18e4ad468de9882e45f2fbe284841b9a99e2e"} Oct 30 18:05:24.590961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:24.590920 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" event={"ID":"22f202a4-66e8-4175-bdab-075f0c0667cb","Type":"ContainerStarted","Data":"d49fa22139cdb8412cd236e754ae75aa7bfbb938e0ca82ad8bc51adb495d0408"} Oct 30 18:05:25.595483 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:25.595445 5232 generic.go:356] "Generic (PLEG): container finished" podID="22f202a4-66e8-4175-bdab-075f0c0667cb" containerID="d49fa22139cdb8412cd236e754ae75aa7bfbb938e0ca82ad8bc51adb495d0408" exitCode=0 Oct 30 18:05:25.595900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:25.595493 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" event={"ID":"22f202a4-66e8-4175-bdab-075f0c0667cb","Type":"ContainerDied","Data":"d49fa22139cdb8412cd236e754ae75aa7bfbb938e0ca82ad8bc51adb495d0408"} Oct 30 18:05:26.601822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:26.601789 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" event={"ID":"22f202a4-66e8-4175-bdab-075f0c0667cb","Type":"ContainerStarted","Data":"504adf8e07a6fbec4f1c28377ddf32ba6a9f432409f510c763e5cdce4dec12a7"} Oct 30 18:05:26.602238 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:26.601913 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:26.686416 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:26.686364 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" podStartSLOduration=2.983979321 podStartE2EDuration="6.68634907s" totalImagesPullingTime="3.234237359s" totalInitContainerRuntime="468.13239ms" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:20 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:26.674148687 +0000 UTC m=+225.990569007" watchObservedRunningTime="2026-10-30 18:05:26.68634907 +0000 UTC m=+226.002769390" Oct 30 18:05:27.032096 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:05:27.032054 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:05:37.608949 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:37.608918 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-85c77bd65b-kt4qd" Oct 30 18:05:54.347461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:54.347426 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh"] Oct 30 18:05:54.349707 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:54.349688 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh" Oct 30 18:05:54.370138 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:54.370118 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 18:05:54.374602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:54.374586 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 18:05:54.376675 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:54.376655 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvz8l\" (UniqueName: \"kubernetes.io/projected/3571d583-d50a-49ff-bf63-ca8eb1823736-kube-api-access-jvz8l\") pod \"kube-auth-proxy-69dd9bcffb-ddqsh\" (UID: \"3571d583-d50a-49ff-bf63-ca8eb1823736\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh" Oct 30 18:05:54.376779 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:54.376691 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3571d583-d50a-49ff-bf63-ca8eb1823736-tls-certs\") pod \"kube-auth-proxy-69dd9bcffb-ddqsh\" (UID: \"3571d583-d50a-49ff-bf63-ca8eb1823736\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh" Oct 30 18:05:54.376879 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:54.376811 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3571d583-d50a-49ff-bf63-ca8eb1823736-tmp\") pod \"kube-auth-proxy-69dd9bcffb-ddqsh\" (UID: \"3571d583-d50a-49ff-bf63-ca8eb1823736\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh" Oct 30 18:05:54.388221 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:54.388197 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh"] Oct 30 18:05:54.428223 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:54.428194 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-nqjgb\"" Oct 30 18:05:54.477758 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:54.477727 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3571d583-d50a-49ff-bf63-ca8eb1823736-tmp\") pod \"kube-auth-proxy-69dd9bcffb-ddqsh\" (UID: \"3571d583-d50a-49ff-bf63-ca8eb1823736\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh" Oct 30 18:05:54.477932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:54.477787 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvz8l\" (UniqueName: \"kubernetes.io/projected/3571d583-d50a-49ff-bf63-ca8eb1823736-kube-api-access-jvz8l\") pod \"kube-auth-proxy-69dd9bcffb-ddqsh\" (UID: \"3571d583-d50a-49ff-bf63-ca8eb1823736\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh" Oct 30 18:05:54.477932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:54.477821 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3571d583-d50a-49ff-bf63-ca8eb1823736-tls-certs\") pod \"kube-auth-proxy-69dd9bcffb-ddqsh\" (UID: \"3571d583-d50a-49ff-bf63-ca8eb1823736\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh" Oct 30 18:05:54.478053 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:05:54.477942 5232 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found Oct 30 18:05:54.478053 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:05:54.478013 5232 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3571d583-d50a-49ff-bf63-ca8eb1823736-tls-certs podName:3571d583-d50a-49ff-bf63-ca8eb1823736 nodeName:}" failed. No retries permitted until 2026-10-30 18:05:54.977992155 +0000 UTC m=+254.294412473 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/3571d583-d50a-49ff-bf63-ca8eb1823736-tls-certs") pod "kube-auth-proxy-69dd9bcffb-ddqsh" (UID: "3571d583-d50a-49ff-bf63-ca8eb1823736") : secret "kube-auth-proxy-tls" not found Oct 30 18:05:54.480112 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:54.480092 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3571d583-d50a-49ff-bf63-ca8eb1823736-tmp\") pod \"kube-auth-proxy-69dd9bcffb-ddqsh\" (UID: \"3571d583-d50a-49ff-bf63-ca8eb1823736\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh" Oct 30 18:05:54.505711 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:54.505679 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvz8l\" (UniqueName: \"kubernetes.io/projected/3571d583-d50a-49ff-bf63-ca8eb1823736-kube-api-access-jvz8l\") pod \"kube-auth-proxy-69dd9bcffb-ddqsh\" (UID: \"3571d583-d50a-49ff-bf63-ca8eb1823736\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh" Oct 30 18:05:54.981795 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:54.981757 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3571d583-d50a-49ff-bf63-ca8eb1823736-tls-certs\") pod \"kube-auth-proxy-69dd9bcffb-ddqsh\" (UID: \"3571d583-d50a-49ff-bf63-ca8eb1823736\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh" Oct 30 18:05:54.984324 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:54.984295 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3571d583-d50a-49ff-bf63-ca8eb1823736-tls-certs\") pod \"kube-auth-proxy-69dd9bcffb-ddqsh\" (UID: \"3571d583-d50a-49ff-bf63-ca8eb1823736\") " pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh" Oct 30 18:05:55.259647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:55.259615 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh" Oct 30 18:05:55.556909 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:55.556827 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh"] Oct 30 18:05:55.561830 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:05:55.561804 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3571d583_d50a_49ff_bf63_ca8eb1823736.slice/crio-440e5fb50432c98e6365481736d366c34a0ba397d2e8b83311830a06e9334634 WatchSource:0}: Error finding container 440e5fb50432c98e6365481736d366c34a0ba397d2e8b83311830a06e9334634: Status 404 returned error can't find the container with id 440e5fb50432c98e6365481736d366c34a0ba397d2e8b83311830a06e9334634 Oct 30 18:05:55.699448 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:55.699410 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh" event={"ID":"3571d583-d50a-49ff-bf63-ca8eb1823736","Type":"ContainerStarted","Data":"440e5fb50432c98e6365481736d366c34a0ba397d2e8b83311830a06e9334634"} Oct 30 18:05:57.037041 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:05:57.036964 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:05:58.711556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:58.711501 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh" event={"ID":"3571d583-d50a-49ff-bf63-ca8eb1823736","Type":"ContainerStarted","Data":"3f8a572e4c07386257a28fd1fdf382a45f7326f57431b53554baa170396ff424"} Oct 30 18:05:58.955172 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:58.955116 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-69dd9bcffb-ddqsh" podStartSLOduration=2.732019051 podStartE2EDuration="4.955099835s" totalImagesPullingTime="2.223080784s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:05:58.763999242 +0000 UTC m=+258.080419562" watchObservedRunningTime="2026-10-30 18:05:58.955099835 +0000 UTC m=+258.271520156" Oct 30 18:05:58.955821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:58.955798 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9kgpx"] Oct 30 18:05:58.959112 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:58.959088 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9kgpx" Oct 30 18:05:58.975631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:58.975606 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-6mbn8\"" Oct 30 18:05:58.983845 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:58.983823 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9kgpx"] Oct 30 18:05:59.011204 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:59.011183 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47c339f9-b26a-41f3-93ce-faa7bfc8dddf-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9kgpx\" (UID: \"47c339f9-b26a-41f3-93ce-faa7bfc8dddf\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9kgpx" Oct 30 18:05:59.011302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:59.011215 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sh9sn\" (UniqueName: \"kubernetes.io/projected/47c339f9-b26a-41f3-93ce-faa7bfc8dddf-kube-api-access-sh9sn\") pod \"cert-manager-cainjector-5c57bbd4fc-9kgpx\" (UID: \"47c339f9-b26a-41f3-93ce-faa7bfc8dddf\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9kgpx" Oct 30 18:05:59.049980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:59.049962 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 18:05:59.050064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:59.049983 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 18:05:59.112231 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:59.112207 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47c339f9-b26a-41f3-93ce-faa7bfc8dddf-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9kgpx\" (UID: \"47c339f9-b26a-41f3-93ce-faa7bfc8dddf\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9kgpx" Oct 30 18:05:59.112324 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:59.112236 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sh9sn\" (UniqueName: \"kubernetes.io/projected/47c339f9-b26a-41f3-93ce-faa7bfc8dddf-kube-api-access-sh9sn\") pod \"cert-manager-cainjector-5c57bbd4fc-9kgpx\" (UID: \"47c339f9-b26a-41f3-93ce-faa7bfc8dddf\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9kgpx" Oct 30 18:05:59.134785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:59.134762 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/47c339f9-b26a-41f3-93ce-faa7bfc8dddf-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9kgpx\" (UID: \"47c339f9-b26a-41f3-93ce-faa7bfc8dddf\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9kgpx" Oct 30 18:05:59.134940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:59.134924 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sh9sn\" (UniqueName: \"kubernetes.io/projected/47c339f9-b26a-41f3-93ce-faa7bfc8dddf-kube-api-access-sh9sn\") pod \"cert-manager-cainjector-5c57bbd4fc-9kgpx\" (UID: \"47c339f9-b26a-41f3-93ce-faa7bfc8dddf\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9kgpx" Oct 30 18:05:59.279327 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:59.279305 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9kgpx" Oct 30 18:05:59.569444 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:59.569380 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9kgpx"] Oct 30 18:05:59.574044 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:05:59.574016 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod47c339f9_b26a_41f3_93ce_faa7bfc8dddf.slice/crio-dfe669e0bd24ad2a2c70613d619fb8fe4cd9fa4a9a2ee40f74895d6cc806a777 WatchSource:0}: Error finding container dfe669e0bd24ad2a2c70613d619fb8fe4cd9fa4a9a2ee40f74895d6cc806a777: Status 404 returned error can't find the container with id dfe669e0bd24ad2a2c70613d619fb8fe4cd9fa4a9a2ee40f74895d6cc806a777 Oct 30 18:05:59.715683 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:05:59.715648 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9kgpx" event={"ID":"47c339f9-b26a-41f3-93ce-faa7bfc8dddf","Type":"ContainerStarted","Data":"dfe669e0bd24ad2a2c70613d619fb8fe4cd9fa4a9a2ee40f74895d6cc806a777"} Oct 30 18:06:01.686644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:01.686584 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-twr9r"] Oct 30 18:06:01.690211 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:01.690190 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-twr9r" Oct 30 18:06:01.714383 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:01.714353 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-tdx78\"" Oct 30 18:06:01.725678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:01.725647 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-twr9r"] Oct 30 18:06:01.734640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:01.734600 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38167f43-0ae8-4157-af62-e8ec5667bd91-bound-sa-token\") pod \"cert-manager-597b95cb84-twr9r\" (UID: \"38167f43-0ae8-4157-af62-e8ec5667bd91\") " pod="cert-manager/cert-manager-597b95cb84-twr9r" Oct 30 18:06:01.734817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:01.734660 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tz697\" (UniqueName: \"kubernetes.io/projected/38167f43-0ae8-4157-af62-e8ec5667bd91-kube-api-access-tz697\") pod \"cert-manager-597b95cb84-twr9r\" (UID: \"38167f43-0ae8-4157-af62-e8ec5667bd91\") " pod="cert-manager/cert-manager-597b95cb84-twr9r" Oct 30 18:06:01.836130 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:01.836089 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38167f43-0ae8-4157-af62-e8ec5667bd91-bound-sa-token\") pod \"cert-manager-597b95cb84-twr9r\" (UID: \"38167f43-0ae8-4157-af62-e8ec5667bd91\") " pod="cert-manager/cert-manager-597b95cb84-twr9r" Oct 30 18:06:01.836130 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:01.836138 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tz697\" (UniqueName: \"kubernetes.io/projected/38167f43-0ae8-4157-af62-e8ec5667bd91-kube-api-access-tz697\") pod \"cert-manager-597b95cb84-twr9r\" (UID: \"38167f43-0ae8-4157-af62-e8ec5667bd91\") " pod="cert-manager/cert-manager-597b95cb84-twr9r" Oct 30 18:06:01.861520 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:01.861480 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/38167f43-0ae8-4157-af62-e8ec5667bd91-bound-sa-token\") pod \"cert-manager-597b95cb84-twr9r\" (UID: \"38167f43-0ae8-4157-af62-e8ec5667bd91\") " pod="cert-manager/cert-manager-597b95cb84-twr9r" Oct 30 18:06:01.862239 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:01.862218 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tz697\" (UniqueName: \"kubernetes.io/projected/38167f43-0ae8-4157-af62-e8ec5667bd91-kube-api-access-tz697\") pod \"cert-manager-597b95cb84-twr9r\" (UID: \"38167f43-0ae8-4157-af62-e8ec5667bd91\") " pod="cert-manager/cert-manager-597b95cb84-twr9r" Oct 30 18:06:02.003950 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:02.003918 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-twr9r" Oct 30 18:06:02.314167 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:02.314055 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-twr9r"] Oct 30 18:06:02.319325 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:06:02.319295 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod38167f43_0ae8_4157_af62_e8ec5667bd91.slice/crio-81a0b502fcb4993c2575b3c1ca624a1def893a83204df1fa1f014dc21a99434a WatchSource:0}: Error finding container 81a0b502fcb4993c2575b3c1ca624a1def893a83204df1fa1f014dc21a99434a: Status 404 returned error can't find the container with id 81a0b502fcb4993c2575b3c1ca624a1def893a83204df1fa1f014dc21a99434a Oct 30 18:06:02.729788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:02.729759 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9kgpx" event={"ID":"47c339f9-b26a-41f3-93ce-faa7bfc8dddf","Type":"ContainerStarted","Data":"9c41ae89bb0d4e654d9e79fe51c3b10faf537b09f7cd0447cafdc107921e7c79"} Oct 30 18:06:02.731352 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:02.731332 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-twr9r" event={"ID":"38167f43-0ae8-4157-af62-e8ec5667bd91","Type":"ContainerStarted","Data":"81a0b502fcb4993c2575b3c1ca624a1def893a83204df1fa1f014dc21a99434a"} Oct 30 18:06:02.776344 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:02.776294 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9kgpx" podStartSLOduration=1.786562166 podStartE2EDuration="4.776278769s" totalImagesPullingTime="2.989716603s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:05:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:02.774881108 +0000 UTC m=+262.091301428" watchObservedRunningTime="2026-10-30 18:06:02.776278769 +0000 UTC m=+262.092699089" Oct 30 18:06:03.736288 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:03.736248 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-twr9r" event={"ID":"38167f43-0ae8-4157-af62-e8ec5667bd91","Type":"ContainerStarted","Data":"19b65868dc8c00eb31d4f522232a84e93b2718883d8b4a45bc859e0de7680d7d"} Oct 30 18:06:03.783715 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:03.783673 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-twr9r" podStartSLOduration=2.426166693 podStartE2EDuration="2.783661437s" totalImagesPullingTime="357.494744ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:03.783086169 +0000 UTC m=+263.099506495" watchObservedRunningTime="2026-10-30 18:06:03.783661437 +0000 UTC m=+263.100081757" Oct 30 18:06:27.032013 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:06:27.031929 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:06:47.948541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:47.948473 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-7g2bf"] Oct 30 18:06:47.953671 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:47.953650 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7g2bf" Oct 30 18:06:47.969729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:47.969705 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-9ppxp\"" Oct 30 18:06:47.982101 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:47.982071 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 30 18:06:47.982302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:47.982124 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 30 18:06:47.982302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:47.982183 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 30 18:06:47.983013 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:47.982988 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-7g2bf"] Oct 30 18:06:47.988617 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:47.988586 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t94dc\" (UniqueName: \"kubernetes.io/projected/edc2fdde-824f-4bd7-950a-34d07337a7bf-kube-api-access-t94dc\") pod \"postgres-7cdbf6dc44-7g2bf\" (UID: \"edc2fdde-824f-4bd7-950a-34d07337a7bf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7g2bf" Oct 30 18:06:47.988617 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:47.988623 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/edc2fdde-824f-4bd7-950a-34d07337a7bf-data\") pod \"postgres-7cdbf6dc44-7g2bf\" (UID: \"edc2fdde-824f-4bd7-950a-34d07337a7bf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7g2bf" Oct 30 18:06:48.089538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:48.089484 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t94dc\" (UniqueName: \"kubernetes.io/projected/edc2fdde-824f-4bd7-950a-34d07337a7bf-kube-api-access-t94dc\") pod \"postgres-7cdbf6dc44-7g2bf\" (UID: \"edc2fdde-824f-4bd7-950a-34d07337a7bf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7g2bf" Oct 30 18:06:48.089720 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:48.089552 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/edc2fdde-824f-4bd7-950a-34d07337a7bf-data\") pod \"postgres-7cdbf6dc44-7g2bf\" (UID: \"edc2fdde-824f-4bd7-950a-34d07337a7bf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7g2bf" Oct 30 18:06:48.089952 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:48.089930 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/edc2fdde-824f-4bd7-950a-34d07337a7bf-data\") pod \"postgres-7cdbf6dc44-7g2bf\" (UID: \"edc2fdde-824f-4bd7-950a-34d07337a7bf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7g2bf" Oct 30 18:06:48.110496 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:48.110459 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t94dc\" (UniqueName: \"kubernetes.io/projected/edc2fdde-824f-4bd7-950a-34d07337a7bf-kube-api-access-t94dc\") pod \"postgres-7cdbf6dc44-7g2bf\" (UID: \"edc2fdde-824f-4bd7-950a-34d07337a7bf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7g2bf" Oct 30 18:06:48.264630 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:48.264595 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7g2bf" Oct 30 18:06:48.553249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:48.553165 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-7g2bf"] Oct 30 18:06:48.558022 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:06:48.557995 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc2fdde_824f_4bd7_950a_34d07337a7bf.slice/crio-0a7b30d638a856b4118d1dcdeb0af5ef50d543f45da03f52ef2e6086fb005d34 WatchSource:0}: Error finding container 0a7b30d638a856b4118d1dcdeb0af5ef50d543f45da03f52ef2e6086fb005d34: Status 404 returned error can't find the container with id 0a7b30d638a856b4118d1dcdeb0af5ef50d543f45da03f52ef2e6086fb005d34 Oct 30 18:06:48.895009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:48.894931 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7g2bf" event={"ID":"edc2fdde-824f-4bd7-950a-34d07337a7bf","Type":"ContainerStarted","Data":"0a7b30d638a856b4118d1dcdeb0af5ef50d543f45da03f52ef2e6086fb005d34"} Oct 30 18:06:53.917141 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:53.917095 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7g2bf" event={"ID":"edc2fdde-824f-4bd7-950a-34d07337a7bf","Type":"ContainerStarted","Data":"9e7f4cbaf3a2f5155bb3cec64bb95b21569bc6ad396682997e819896e0f9c790"} Oct 30 18:06:53.917561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:53.917281 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7g2bf" Oct 30 18:06:53.964037 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:53.963990 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7g2bf" podStartSLOduration=2.082911418 podStartE2EDuration="6.96397496s" totalImagesPullingTime="4.881063542s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:47 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:53.963254515 +0000 UTC m=+313.279674838" watchObservedRunningTime="2026-10-30 18:06:53.96397496 +0000 UTC m=+313.280395282" Oct 30 18:06:57.031282 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:06:57.031240 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:06:57.237843 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.237788 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp"] Oct 30 18:06:57.241256 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.241235 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.269312 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.269274 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp"] Oct 30 18:06:57.269604 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.269584 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.269690 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.269668 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.269743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.269707 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.269789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.269752 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322b3612-668d-4d50-8506-9df154d5e57d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.269856 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.269831 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/322b3612-668d-4d50-8506-9df154d5e57d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.269910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.269878 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/322b3612-668d-4d50-8506-9df154d5e57d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.269949 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.269926 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qt4fm\" (UniqueName: \"kubernetes.io/projected/322b3612-668d-4d50-8506-9df154d5e57d-kube-api-access-qt4fm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.371261 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.371175 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.371261 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.371213 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.371261 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.371249 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322b3612-668d-4d50-8506-9df154d5e57d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.371576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.371284 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/322b3612-668d-4d50-8506-9df154d5e57d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.371576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.371301 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/322b3612-668d-4d50-8506-9df154d5e57d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.371576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.371315 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qt4fm\" (UniqueName: \"kubernetes.io/projected/322b3612-668d-4d50-8506-9df154d5e57d-kube-api-access-qt4fm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.371576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.371364 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.372337 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.372303 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/322b3612-668d-4d50-8506-9df154d5e57d-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.373844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.373820 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.373958 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.373892 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/322b3612-668d-4d50-8506-9df154d5e57d-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.373958 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.373932 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.373958 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.373946 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.393711 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.393687 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qt4fm\" (UniqueName: \"kubernetes.io/projected/322b3612-668d-4d50-8506-9df154d5e57d-kube-api-access-qt4fm\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.397239 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.397217 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322b3612-668d-4d50-8506-9df154d5e57d-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-t7ntp\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.552464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.552428 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.858412 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.858375 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp"] Oct 30 18:06:57.864542 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:06:57.864482 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod322b3612_668d_4d50_8506_9df154d5e57d.slice/crio-278e90aae64dbcfec5ef1203757e5ca65950092357c28f7a41e51c717eba54b2 WatchSource:0}: Error finding container 278e90aae64dbcfec5ef1203757e5ca65950092357c28f7a41e51c717eba54b2: Status 404 returned error can't find the container with id 278e90aae64dbcfec5ef1203757e5ca65950092357c28f7a41e51c717eba54b2 Oct 30 18:06:57.866859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.866826 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:06:57.866940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.866889 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:06:57.934953 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.934923 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" event={"ID":"322b3612-668d-4d50-8506-9df154d5e57d","Type":"ContainerStarted","Data":"4765120e0dbf004d1b47886aac1e935164e16ae8a81bd66c9e50b1dbe2f86f41"} Oct 30 18:06:57.935092 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.934971 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" event={"ID":"322b3612-668d-4d50-8506-9df154d5e57d","Type":"ContainerStarted","Data":"278e90aae64dbcfec5ef1203757e5ca65950092357c28f7a41e51c717eba54b2"} Oct 30 18:06:57.935092 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.935044 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:06:57.979399 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:57.979344 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" podStartSLOduration=0.979329363 podStartE2EDuration="979.329363ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:06:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:06:57.978282586 +0000 UTC m=+317.294702906" watchObservedRunningTime="2026-10-30 18:06:57.979329363 +0000 UTC m=+317.295749684" Oct 30 18:06:58.941288 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:58.941256 5232 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-t7ntp container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:06:58.941757 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:58.941619 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" podUID="322b3612-668d-4d50-8506-9df154d5e57d" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:06:59.949605 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:06:59.949578 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-7g2bf" Oct 30 18:07:01.940224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:01.940195 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:07:02.025169 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.025137 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg"] Oct 30 18:07:02.025389 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.025359 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" podUID="09bbbf57-61a9-4403-a805-b86b03119b97" containerName="discovery" containerID="cri-o://dce7618de54b80b24e4a8f8df4c521c5a8ae4fcd8faf89a9b81394ea8b8b765c" gracePeriod=30 Oct 30 18:07:02.279079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.279046 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:07:02.312324 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.312293 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/09bbbf57-61a9-4403-a805-b86b03119b97-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bbbf57-61a9-4403-a805-b86b03119b97-local-certs\") pod \"09bbbf57-61a9-4403-a805-b86b03119b97\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " Oct 30 18:07:02.312479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.312333 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09bbbf57-61a9-4403-a805-b86b03119b97-kube-api-access-r74fk\" (UniqueName: \"kubernetes.io/projected/09bbbf57-61a9-4403-a805-b86b03119b97-kube-api-access-r74fk\") pod \"09bbbf57-61a9-4403-a805-b86b03119b97\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " Oct 30 18:07:02.312479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.312370 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-cacerts\" (UniqueName: \"kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-cacerts\") pod \"09bbbf57-61a9-4403-a805-b86b03119b97\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " Oct 30 18:07:02.312479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.312408 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-istio-csr-dns-cert\") pod \"09bbbf57-61a9-4403-a805-b86b03119b97\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " Oct 30 18:07:02.312479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.312451 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-istio-kubeconfig\") pod \"09bbbf57-61a9-4403-a805-b86b03119b97\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " Oct 30 18:07:02.312712 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.312524 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/09bbbf57-61a9-4403-a805-b86b03119b97-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/09bbbf57-61a9-4403-a805-b86b03119b97-istio-csr-ca-configmap\") pod \"09bbbf57-61a9-4403-a805-b86b03119b97\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " Oct 30 18:07:02.312712 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.312561 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09bbbf57-61a9-4403-a805-b86b03119b97-istio-token\" (UniqueName: \"kubernetes.io/projected/09bbbf57-61a9-4403-a805-b86b03119b97-istio-token\") pod \"09bbbf57-61a9-4403-a805-b86b03119b97\" (UID: \"09bbbf57-61a9-4403-a805-b86b03119b97\") " Oct 30 18:07:02.312970 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.312930 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09bbbf57-61a9-4403-a805-b86b03119b97-istio-csr-ca-configmap" pod "09bbbf57-61a9-4403-a805-b86b03119b97" (UID: "09bbbf57-61a9-4403-a805-b86b03119b97"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:07:02.314938 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.314913 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09bbbf57-61a9-4403-a805-b86b03119b97-kube-api-access-r74fk" pod "09bbbf57-61a9-4403-a805-b86b03119b97" (UID: "09bbbf57-61a9-4403-a805-b86b03119b97"). InnerVolumeSpecName "kube-api-access-r74fk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:02.315058 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.315039 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-istio-kubeconfig" pod "09bbbf57-61a9-4403-a805-b86b03119b97" (UID: "09bbbf57-61a9-4403-a805-b86b03119b97"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:07:02.315125 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.315110 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-cacerts" pod "09bbbf57-61a9-4403-a805-b86b03119b97" (UID: "09bbbf57-61a9-4403-a805-b86b03119b97"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:07:02.315187 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.315134 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-istio-csr-dns-cert" pod "09bbbf57-61a9-4403-a805-b86b03119b97" (UID: "09bbbf57-61a9-4403-a805-b86b03119b97"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:07:02.315240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.315199 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09bbbf57-61a9-4403-a805-b86b03119b97-local-certs" pod "09bbbf57-61a9-4403-a805-b86b03119b97" (UID: "09bbbf57-61a9-4403-a805-b86b03119b97"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:07:02.315401 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.315380 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09bbbf57-61a9-4403-a805-b86b03119b97-istio-token" pod "09bbbf57-61a9-4403-a805-b86b03119b97" (UID: "09bbbf57-61a9-4403-a805-b86b03119b97"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:07:02.413900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.413864 5232 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/09bbbf57-61a9-4403-a805-b86b03119b97-istio-csr-ca-configmap\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:07:02.413900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.413895 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/09bbbf57-61a9-4403-a805-b86b03119b97-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:07:02.413900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.413905 5232 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/09bbbf57-61a9-4403-a805-b86b03119b97-local-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:07:02.414136 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.413915 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r74fk\" (UniqueName: \"kubernetes.io/projected/09bbbf57-61a9-4403-a805-b86b03119b97-kube-api-access-r74fk\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:07:02.414136 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.413924 5232 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-cacerts\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:07:02.414136 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.413933 5232 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-istio-csr-dns-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:07:02.414136 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.413942 5232 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/09bbbf57-61a9-4403-a805-b86b03119b97-istio-kubeconfig\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:07:02.954668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.954629 5232 generic.go:356] "Generic (PLEG): container finished" podID="09bbbf57-61a9-4403-a805-b86b03119b97" containerID="dce7618de54b80b24e4a8f8df4c521c5a8ae4fcd8faf89a9b81394ea8b8b765c" exitCode=0 Oct 30 18:07:02.955118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.954681 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" Oct 30 18:07:02.955118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.954736 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" event={"ID":"09bbbf57-61a9-4403-a805-b86b03119b97","Type":"ContainerDied","Data":"dce7618de54b80b24e4a8f8df4c521c5a8ae4fcd8faf89a9b81394ea8b8b765c"} Oct 30 18:07:02.955118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.954785 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg" event={"ID":"09bbbf57-61a9-4403-a805-b86b03119b97","Type":"ContainerDied","Data":"78a9dc7b24f11fc62dbe0f044abcdfde639767ed0d3122bedb70d315252278d6"} Oct 30 18:07:02.955118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.954807 5232 scope.go:122] "RemoveContainer" containerID="dce7618de54b80b24e4a8f8df4c521c5a8ae4fcd8faf89a9b81394ea8b8b765c" Oct 30 18:07:02.962831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.962814 5232 scope.go:122] "RemoveContainer" containerID="dce7618de54b80b24e4a8f8df4c521c5a8ae4fcd8faf89a9b81394ea8b8b765c" Oct 30 18:07:02.963102 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:07:02.963085 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dce7618de54b80b24e4a8f8df4c521c5a8ae4fcd8faf89a9b81394ea8b8b765c\": container with ID starting with dce7618de54b80b24e4a8f8df4c521c5a8ae4fcd8faf89a9b81394ea8b8b765c not found: ID does not exist" containerID="dce7618de54b80b24e4a8f8df4c521c5a8ae4fcd8faf89a9b81394ea8b8b765c" Oct 30 18:07:02.963165 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:02.963115 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dce7618de54b80b24e4a8f8df4c521c5a8ae4fcd8faf89a9b81394ea8b8b765c"} err="failed to get container status \"dce7618de54b80b24e4a8f8df4c521c5a8ae4fcd8faf89a9b81394ea8b8b765c\": rpc error: code = NotFound desc = could not find container \"dce7618de54b80b24e4a8f8df4c521c5a8ae4fcd8faf89a9b81394ea8b8b765c\": container with ID starting with dce7618de54b80b24e4a8f8df4c521c5a8ae4fcd8faf89a9b81394ea8b8b765c not found: ID does not exist" Oct 30 18:07:03.020672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:03.020636 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg"] Oct 30 18:07:03.032906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:03.032871 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-7p9tg"] Oct 30 18:07:04.800012 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:04.799978 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09bbbf57-61a9-4403-a805-b86b03119b97" path="/var/lib/kubelet/pods/09bbbf57-61a9-4403-a805-b86b03119b97/volumes" Oct 30 18:07:07.482637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.482603 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68c97f9884-dsjzk"] Oct 30 18:07:07.483055 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.482889 5232 cpu_manager.go:414] "containerMap: removing container" podUID="09bbbf57-61a9-4403-a805-b86b03119b97" containerName="discovery" Oct 30 18:07:07.483055 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.482899 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09bbbf57-61a9-4403-a805-b86b03119b97" containerName="discovery" Oct 30 18:07:07.483055 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.482957 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="09bbbf57-61a9-4403-a805-b86b03119b97" containerName="discovery" Oct 30 18:07:07.504102 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.504072 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c97f9884-dsjzk" Oct 30 18:07:07.512487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.512461 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c97f9884-dsjzk"] Oct 30 18:07:07.523226 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.523197 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 18:07:07.523226 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.523222 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 18:07:07.523399 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.523197 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 18:07:07.531617 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.531598 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 18:07:07.531617 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.531599 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-r2cqt\"" Oct 30 18:07:07.531756 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.531636 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 18:07:07.555152 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.555125 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f76e41be-f30a-4c44-90cc-ef0b08127828-webhook-cert\") pod \"maas-controller-68c97f9884-dsjzk\" (UID: \"f76e41be-f30a-4c44-90cc-ef0b08127828\") " pod="opendatahub/maas-controller-68c97f9884-dsjzk" Oct 30 18:07:07.555260 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.555179 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76e41be-f30a-4c44-90cc-ef0b08127828-metrics-certs\") pod \"maas-controller-68c97f9884-dsjzk\" (UID: \"f76e41be-f30a-4c44-90cc-ef0b08127828\") " pod="opendatahub/maas-controller-68c97f9884-dsjzk" Oct 30 18:07:07.555260 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.555236 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdcd7\" (UniqueName: \"kubernetes.io/projected/f76e41be-f30a-4c44-90cc-ef0b08127828-kube-api-access-jdcd7\") pod \"maas-controller-68c97f9884-dsjzk\" (UID: \"f76e41be-f30a-4c44-90cc-ef0b08127828\") " pod="opendatahub/maas-controller-68c97f9884-dsjzk" Oct 30 18:07:07.656080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.656049 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76e41be-f30a-4c44-90cc-ef0b08127828-metrics-certs\") pod \"maas-controller-68c97f9884-dsjzk\" (UID: \"f76e41be-f30a-4c44-90cc-ef0b08127828\") " pod="opendatahub/maas-controller-68c97f9884-dsjzk" Oct 30 18:07:07.656249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.656088 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdcd7\" (UniqueName: \"kubernetes.io/projected/f76e41be-f30a-4c44-90cc-ef0b08127828-kube-api-access-jdcd7\") pod \"maas-controller-68c97f9884-dsjzk\" (UID: \"f76e41be-f30a-4c44-90cc-ef0b08127828\") " pod="opendatahub/maas-controller-68c97f9884-dsjzk" Oct 30 18:07:07.656249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.656120 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f76e41be-f30a-4c44-90cc-ef0b08127828-webhook-cert\") pod \"maas-controller-68c97f9884-dsjzk\" (UID: \"f76e41be-f30a-4c44-90cc-ef0b08127828\") " pod="opendatahub/maas-controller-68c97f9884-dsjzk" Oct 30 18:07:07.658636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.658612 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76e41be-f30a-4c44-90cc-ef0b08127828-metrics-certs\") pod \"maas-controller-68c97f9884-dsjzk\" (UID: \"f76e41be-f30a-4c44-90cc-ef0b08127828\") " pod="opendatahub/maas-controller-68c97f9884-dsjzk" Oct 30 18:07:07.658773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.658752 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f76e41be-f30a-4c44-90cc-ef0b08127828-webhook-cert\") pod \"maas-controller-68c97f9884-dsjzk\" (UID: \"f76e41be-f30a-4c44-90cc-ef0b08127828\") " pod="opendatahub/maas-controller-68c97f9884-dsjzk" Oct 30 18:07:07.675726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.675693 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdcd7\" (UniqueName: \"kubernetes.io/projected/f76e41be-f30a-4c44-90cc-ef0b08127828-kube-api-access-jdcd7\") pod \"maas-controller-68c97f9884-dsjzk\" (UID: \"f76e41be-f30a-4c44-90cc-ef0b08127828\") " pod="opendatahub/maas-controller-68c97f9884-dsjzk" Oct 30 18:07:07.817164 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:07.817131 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c97f9884-dsjzk" Oct 30 18:07:08.108351 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:08.108267 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c97f9884-dsjzk"] Oct 30 18:07:08.112767 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:07:08.112738 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf76e41be_f30a_4c44_90cc_ef0b08127828.slice/crio-adb0d938d6816669157a767739395a21319d7ac9e3afeffc792c58d1369253c5 WatchSource:0}: Error finding container adb0d938d6816669157a767739395a21319d7ac9e3afeffc792c58d1369253c5: Status 404 returned error can't find the container with id adb0d938d6816669157a767739395a21319d7ac9e3afeffc792c58d1369253c5 Oct 30 18:07:08.980712 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:08.980669 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c97f9884-dsjzk" event={"ID":"f76e41be-f30a-4c44-90cc-ef0b08127828","Type":"ContainerStarted","Data":"adb0d938d6816669157a767739395a21319d7ac9e3afeffc792c58d1369253c5"} Oct 30 18:07:10.991240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:10.991197 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c97f9884-dsjzk" event={"ID":"f76e41be-f30a-4c44-90cc-ef0b08127828","Type":"ContainerStarted","Data":"d5d32e7f2f8775dcbbc28f3ca718e746c9e85675da59073fd3b28667b8b0b5e8"} Oct 30 18:07:10.991813 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:10.991308 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68c97f9884-dsjzk" Oct 30 18:07:11.036470 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.036411 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-68c97f9884-dsjzk" podStartSLOduration=1.745475389 podStartE2EDuration="4.036391115s" totalImagesPullingTime="2.290915726s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:07 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:11.034661888 +0000 UTC m=+330.351082283" watchObservedRunningTime="2026-10-30 18:07:11.036391115 +0000 UTC m=+330.352811437" Oct 30 18:07:11.504274 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.504236 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5"] Oct 30 18:07:11.507650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.507626 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" Oct 30 18:07:11.539469 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.539437 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" Oct 30 18:07:11.539726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.539478 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" Oct 30 18:07:11.539726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.539479 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" Oct 30 18:07:11.539726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.539478 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-d9wpc\"" Oct 30 18:07:11.539726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.539489 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" Oct 30 18:07:11.540033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.539491 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" Oct 30 18:07:11.540630 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.540608 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5"] Oct 30 18:07:11.592594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.592565 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a171996f-f034-4ee7-bdb8-329d0f42864c-manager-config\") pod \"jobset-controller-manager-dcbbb97bd-v7cq5\" (UID: \"a171996f-f034-4ee7-bdb8-329d0f42864c\") " pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" Oct 30 18:07:11.592747 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.592612 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a171996f-f034-4ee7-bdb8-329d0f42864c-metrics-certs\") pod \"jobset-controller-manager-dcbbb97bd-v7cq5\" (UID: \"a171996f-f034-4ee7-bdb8-329d0f42864c\") " pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" Oct 30 18:07:11.592747 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.592643 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a171996f-f034-4ee7-bdb8-329d0f42864c-cert\") pod \"jobset-controller-manager-dcbbb97bd-v7cq5\" (UID: \"a171996f-f034-4ee7-bdb8-329d0f42864c\") " pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" Oct 30 18:07:11.592834 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.592813 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmktp\" (UniqueName: \"kubernetes.io/projected/a171996f-f034-4ee7-bdb8-329d0f42864c-kube-api-access-rmktp\") pod \"jobset-controller-manager-dcbbb97bd-v7cq5\" (UID: \"a171996f-f034-4ee7-bdb8-329d0f42864c\") " pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" Oct 30 18:07:11.694079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.694044 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmktp\" (UniqueName: \"kubernetes.io/projected/a171996f-f034-4ee7-bdb8-329d0f42864c-kube-api-access-rmktp\") pod \"jobset-controller-manager-dcbbb97bd-v7cq5\" (UID: \"a171996f-f034-4ee7-bdb8-329d0f42864c\") " pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" Oct 30 18:07:11.694286 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.694090 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a171996f-f034-4ee7-bdb8-329d0f42864c-manager-config\") pod \"jobset-controller-manager-dcbbb97bd-v7cq5\" (UID: \"a171996f-f034-4ee7-bdb8-329d0f42864c\") " pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" Oct 30 18:07:11.694286 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.694118 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a171996f-f034-4ee7-bdb8-329d0f42864c-metrics-certs\") pod \"jobset-controller-manager-dcbbb97bd-v7cq5\" (UID: \"a171996f-f034-4ee7-bdb8-329d0f42864c\") " pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" Oct 30 18:07:11.694286 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.694138 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a171996f-f034-4ee7-bdb8-329d0f42864c-cert\") pod \"jobset-controller-manager-dcbbb97bd-v7cq5\" (UID: \"a171996f-f034-4ee7-bdb8-329d0f42864c\") " pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" Oct 30 18:07:11.694881 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.694850 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/a171996f-f034-4ee7-bdb8-329d0f42864c-manager-config\") pod \"jobset-controller-manager-dcbbb97bd-v7cq5\" (UID: \"a171996f-f034-4ee7-bdb8-329d0f42864c\") " pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" Oct 30 18:07:11.696896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.696876 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a171996f-f034-4ee7-bdb8-329d0f42864c-metrics-certs\") pod \"jobset-controller-manager-dcbbb97bd-v7cq5\" (UID: \"a171996f-f034-4ee7-bdb8-329d0f42864c\") " pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" Oct 30 18:07:11.696980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.696933 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/a171996f-f034-4ee7-bdb8-329d0f42864c-cert\") pod \"jobset-controller-manager-dcbbb97bd-v7cq5\" (UID: \"a171996f-f034-4ee7-bdb8-329d0f42864c\") " pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" Oct 30 18:07:11.713983 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.713951 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmktp\" (UniqueName: \"kubernetes.io/projected/a171996f-f034-4ee7-bdb8-329d0f42864c-kube-api-access-rmktp\") pod \"jobset-controller-manager-dcbbb97bd-v7cq5\" (UID: \"a171996f-f034-4ee7-bdb8-329d0f42864c\") " pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" Oct 30 18:07:11.819659 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:11.819553 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" Oct 30 18:07:12.126191 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:12.126106 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5"] Oct 30 18:07:12.130178 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:07:12.130151 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda171996f_f034_4ee7_bdb8_329d0f42864c.slice/crio-f8a995df7e018357b32353f072c0d1abe83b069f1ffcd3dbad18ed3f8c0dcbd7 WatchSource:0}: Error finding container f8a995df7e018357b32353f072c0d1abe83b069f1ffcd3dbad18ed3f8c0dcbd7: Status 404 returned error can't find the container with id f8a995df7e018357b32353f072c0d1abe83b069f1ffcd3dbad18ed3f8c0dcbd7 Oct 30 18:07:12.132138 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:12.132118 5232 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:07:13.001327 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:13.001288 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" event={"ID":"a171996f-f034-4ee7-bdb8-329d0f42864c","Type":"ContainerStarted","Data":"f8a995df7e018357b32353f072c0d1abe83b069f1ffcd3dbad18ed3f8c0dcbd7"} Oct 30 18:07:15.011164 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:15.011135 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" event={"ID":"a171996f-f034-4ee7-bdb8-329d0f42864c","Type":"ContainerStarted","Data":"b3db12d514b5413d8641da8a3b72f49b06197e85f9b90b19bb233d22be2bb4a5"} Oct 30 18:07:15.011554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:15.011251 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" Oct 30 18:07:15.051006 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:15.050923 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" podStartSLOduration=1.298210095 podStartE2EDuration="4.050904553s" totalImagesPullingTime="2.752694458s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:07:15.050091381 +0000 UTC m=+334.366511702" watchObservedRunningTime="2026-10-30 18:07:15.050904553 +0000 UTC m=+334.367324875" Oct 30 18:07:21.997669 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:21.997634 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68c97f9884-dsjzk" Oct 30 18:07:26.020438 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:26.020406 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-dcbbb97bd-v7cq5" Oct 30 18:07:27.032875 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:07:27.032841 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:07:57.032967 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:07:57.032886 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:07:58.103286 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.103253 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt"] Oct 30 18:07:58.108459 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.108420 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.130887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.130863 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt"] Oct 30 18:07:58.179621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.179590 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.179792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.179684 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.179792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.179722 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea67986-73d8-4a9d-9268-a664b61c099c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.179792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.179750 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.179792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.179767 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.179792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.179782 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.180051 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.179984 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.180051 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.180017 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.180146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.180058 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jg6p7\" (UniqueName: \"kubernetes.io/projected/0ea67986-73d8-4a9d-9268-a664b61c099c-kube-api-access-jg6p7\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.208765 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.208737 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-jf5s7\"" Oct 30 18:07:58.280477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.280445 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.280477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.280483 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea67986-73d8-4a9d-9268-a664b61c099c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.280739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.280529 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.280739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.280649 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.280739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.280687 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.281139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.280743 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.281139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.280758 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.281139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.280791 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jg6p7\" (UniqueName: \"kubernetes.io/projected/0ea67986-73d8-4a9d-9268-a664b61c099c-kube-api-access-jg6p7\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.281139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.280828 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.281139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.281075 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.281580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.281166 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.281580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.281251 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.281580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.281482 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.281744 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.281667 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea67986-73d8-4a9d-9268-a664b61c099c-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.283131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.283113 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.283248 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.283231 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.300912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.300888 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.301087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.301066 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jg6p7\" (UniqueName: \"kubernetes.io/projected/0ea67986-73d8-4a9d-9268-a664b61c099c-kube-api-access-jg6p7\") pod \"maas-default-gateway-openshift-default-77f556d647-cxtkt\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.423410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.423349 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:07:58.727520 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:58.727426 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt"] Oct 30 18:07:58.731454 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:07:58.731430 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea67986_73d8_4a9d_9268_a664b61c099c.slice/crio-0235ea435e7c95f5e8ffe39b21ced50e0363cf7d6f5ef4f3fba5da2a0a70bab0 WatchSource:0}: Error finding container 0235ea435e7c95f5e8ffe39b21ced50e0363cf7d6f5ef4f3fba5da2a0a70bab0: Status 404 returned error can't find the container with id 0235ea435e7c95f5e8ffe39b21ced50e0363cf7d6f5ef4f3fba5da2a0a70bab0 Oct 30 18:07:59.179576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:07:59.179535 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" event={"ID":"0ea67986-73d8-4a9d-9268-a664b61c099c","Type":"ContainerStarted","Data":"0235ea435e7c95f5e8ffe39b21ced50e0363cf7d6f5ef4f3fba5da2a0a70bab0"} Oct 30 18:08:01.411208 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:01.411172 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:08:01.411455 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:01.411251 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:08:01.411455 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:01.411279 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:08:02.193918 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:02.193879 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" event={"ID":"0ea67986-73d8-4a9d-9268-a664b61c099c","Type":"ContainerStarted","Data":"dbe62416218fcc727e819bf7f386840897dcb415893c5624a6bac7a2704bfc5a"} Oct 30 18:08:02.261097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:02.261042 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" podStartSLOduration=1.584153784 podStartE2EDuration="4.261026359s" totalImagesPullingTime="2.676872575s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:07:58 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:02.260182179 +0000 UTC m=+381.576602500" watchObservedRunningTime="2026-10-30 18:08:02.261026359 +0000 UTC m=+381.577446679" Oct 30 18:08:02.430625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:02.430581 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:08:02.431658 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:02.431627 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cxtkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Oct 30 18:08:02.431855 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:02.431828 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" podUID="0ea67986-73d8-4a9d-9268-a664b61c099c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Oct 30 18:08:03.431062 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:03.431023 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cxtkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Oct 30 18:08:03.431740 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:03.431696 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" podUID="0ea67986-73d8-4a9d-9268-a664b61c099c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Oct 30 18:08:04.430544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:04.429965 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cxtkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Oct 30 18:08:04.430544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:04.430032 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" podUID="0ea67986-73d8-4a9d-9268-a664b61c099c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Oct 30 18:08:05.429023 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:05.428985 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cxtkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Oct 30 18:08:05.429687 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:05.429645 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" podUID="0ea67986-73d8-4a9d-9268-a664b61c099c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Oct 30 18:08:06.429967 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:06.429928 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cxtkt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" start-of-body= Oct 30 18:08:06.430611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:06.430578 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" podUID="0ea67986-73d8-4a9d-9268-a664b61c099c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": dial tcp 10.133.0.23:15021: connect: connection refused" Oct 30 18:08:07.427928 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:07.427899 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:08:07.428196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:07.428174 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:08:07.428844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:07.428826 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:08:27.031360 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:08:27.031321 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:08:28.177683 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.177628 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb"] Oct 30 18:08:28.186129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.186106 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.209307 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.209281 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb"] Oct 30 18:08:28.360207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.360174 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59d61286-680e-4bd9-9e35-1127f589f8f7-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.360207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.360210 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d61286-680e-4bd9-9e35-1127f589f8f7-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.360404 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.360239 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59d61286-680e-4bd9-9e35-1127f589f8f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.360404 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.360352 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6fmn\" (UniqueName: \"kubernetes.io/projected/59d61286-680e-4bd9-9e35-1127f589f8f7-kube-api-access-q6fmn\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.360404 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.360392 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d61286-680e-4bd9-9e35-1127f589f8f7-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.360501 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.360412 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59d61286-680e-4bd9-9e35-1127f589f8f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.360562 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.360524 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d61286-680e-4bd9-9e35-1127f589f8f7-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.360595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.360559 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d61286-680e-4bd9-9e35-1127f589f8f7-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.360595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.360583 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59d61286-680e-4bd9-9e35-1127f589f8f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.461619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.461545 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59d61286-680e-4bd9-9e35-1127f589f8f7-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.461619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.461584 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d61286-680e-4bd9-9e35-1127f589f8f7-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.461619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.461608 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59d61286-680e-4bd9-9e35-1127f589f8f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.461877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.461631 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6fmn\" (UniqueName: \"kubernetes.io/projected/59d61286-680e-4bd9-9e35-1127f589f8f7-kube-api-access-q6fmn\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.461877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.461657 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d61286-680e-4bd9-9e35-1127f589f8f7-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.461877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.461672 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59d61286-680e-4bd9-9e35-1127f589f8f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.461877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.461689 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d61286-680e-4bd9-9e35-1127f589f8f7-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.461877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.461715 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d61286-680e-4bd9-9e35-1127f589f8f7-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.461877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.461740 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59d61286-680e-4bd9-9e35-1127f589f8f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.462177 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.461976 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/59d61286-680e-4bd9-9e35-1127f589f8f7-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.462177 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.462045 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/59d61286-680e-4bd9-9e35-1127f589f8f7-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.462177 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.462120 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d61286-680e-4bd9-9e35-1127f589f8f7-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.462339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.462319 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/59d61286-680e-4bd9-9e35-1127f589f8f7-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.462402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.462383 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/59d61286-680e-4bd9-9e35-1127f589f8f7-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.464078 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.464057 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/59d61286-680e-4bd9-9e35-1127f589f8f7-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.464381 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.464355 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/59d61286-680e-4bd9-9e35-1127f589f8f7-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.481367 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.481337 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/59d61286-680e-4bd9-9e35-1127f589f8f7-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.481566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.481546 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6fmn\" (UniqueName: \"kubernetes.io/projected/59d61286-680e-4bd9-9e35-1127f589f8f7-kube-api-access-q6fmn\") pod \"maas-default-gateway-openshift-default-77f556d647-pxcpb\" (UID: \"59d61286-680e-4bd9-9e35-1127f589f8f7\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.498896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.498875 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:28.808049 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.808018 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb"] Oct 30 18:08:28.812838 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:08:28.812804 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59d61286_680e_4bd9_9e35_1127f589f8f7.slice/crio-c65542c6549b839a24f84ec849920d9ebd1e18e461fe16e21514c7ea6eda505c WatchSource:0}: Error finding container c65542c6549b839a24f84ec849920d9ebd1e18e461fe16e21514c7ea6eda505c: Status 404 returned error can't find the container with id c65542c6549b839a24f84ec849920d9ebd1e18e461fe16e21514c7ea6eda505c Oct 30 18:08:28.814879 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.814842 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:08:28.814950 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.814913 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:08:28.814950 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:28.814943 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:08:29.335692 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:29.335649 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" event={"ID":"59d61286-680e-4bd9-9e35-1127f589f8f7","Type":"ContainerStarted","Data":"b5ab574a25cbccea2ef3b98223b4a78a041abfa61802f439e3dd6fb11bde5cee"} Oct 30 18:08:29.336109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:29.335715 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" event={"ID":"59d61286-680e-4bd9-9e35-1127f589f8f7","Type":"ContainerStarted","Data":"c65542c6549b839a24f84ec849920d9ebd1e18e461fe16e21514c7ea6eda505c"} Oct 30 18:08:29.402938 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:29.402883 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" podStartSLOduration=1.40286653 podStartE2EDuration="1.40286653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:08:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:08:29.389815597 +0000 UTC m=+408.706235919" watchObservedRunningTime="2026-10-30 18:08:29.40286653 +0000 UTC m=+408.719286850" Oct 30 18:08:29.499467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:29.499437 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:29.500885 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:29.500857 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-pxcpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Oct 30 18:08:29.501005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:29.500911 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" podUID="59d61286-680e-4bd9-9e35-1127f589f8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Oct 30 18:08:30.500193 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:30.500156 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-pxcpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Oct 30 18:08:30.500607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:30.500209 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" podUID="59d61286-680e-4bd9-9e35-1127f589f8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Oct 30 18:08:31.500191 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:31.500152 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-pxcpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Oct 30 18:08:31.500621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:31.500222 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" podUID="59d61286-680e-4bd9-9e35-1127f589f8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Oct 30 18:08:32.499922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:32.499892 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-pxcpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Oct 30 18:08:32.500190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:32.499946 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" podUID="59d61286-680e-4bd9-9e35-1127f589f8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Oct 30 18:08:33.499479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:33.499436 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-pxcpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Oct 30 18:08:33.499910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:33.499492 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" podUID="59d61286-680e-4bd9-9e35-1127f589f8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Oct 30 18:08:34.500226 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:34.500186 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-pxcpb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" start-of-body= Oct 30 18:08:34.500734 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:34.500262 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" podUID="59d61286-680e-4bd9-9e35-1127f589f8f7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.24:15021/healthz/ready\": dial tcp 10.133.0.24:15021: connect: connection refused" Oct 30 18:08:35.503102 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:35.503067 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:35.503550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:35.503343 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:35.504099 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:08:35.504083 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-pxcpb" Oct 30 18:08:57.082102 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:08:57.082061 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:09:02.530207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.530172 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr"] Oct 30 18:09:02.533874 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.533608 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.548581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.548557 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 30 18:09:02.554923 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.554891 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b2b3e72-d4ac-4407-97ae-421c5ad091a1-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7k4cr\" (UID: \"1b2b3e72-d4ac-4407-97ae-421c5ad091a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.555502 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.555480 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b2b3e72-d4ac-4407-97ae-421c5ad091a1-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7k4cr\" (UID: \"1b2b3e72-d4ac-4407-97ae-421c5ad091a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.555686 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.555670 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b2b3e72-d4ac-4407-97ae-421c5ad091a1-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7k4cr\" (UID: \"1b2b3e72-d4ac-4407-97ae-421c5ad091a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.555817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.555792 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbmst\" (UniqueName: \"kubernetes.io/projected/1b2b3e72-d4ac-4407-97ae-421c5ad091a1-kube-api-access-qbmst\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7k4cr\" (UID: \"1b2b3e72-d4ac-4407-97ae-421c5ad091a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.555960 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.555871 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b2b3e72-d4ac-4407-97ae-421c5ad091a1-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7k4cr\" (UID: \"1b2b3e72-d4ac-4407-97ae-421c5ad091a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.555960 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.555895 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b2b3e72-d4ac-4407-97ae-421c5ad091a1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7k4cr\" (UID: \"1b2b3e72-d4ac-4407-97ae-421c5ad091a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.560827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.560798 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 30 18:09:02.560990 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.560973 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-embedding-simulated-kserve-self-signed-certs\"" Oct 30 18:09:02.561150 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.561135 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-8g42t\"" Oct 30 18:09:02.562782 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.562761 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr"] Oct 30 18:09:02.656738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.656702 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b2b3e72-d4ac-4407-97ae-421c5ad091a1-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7k4cr\" (UID: \"1b2b3e72-d4ac-4407-97ae-421c5ad091a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.656926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.656748 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b2b3e72-d4ac-4407-97ae-421c5ad091a1-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7k4cr\" (UID: \"1b2b3e72-d4ac-4407-97ae-421c5ad091a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.656926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.656775 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b2b3e72-d4ac-4407-97ae-421c5ad091a1-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7k4cr\" (UID: \"1b2b3e72-d4ac-4407-97ae-421c5ad091a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.656926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.656801 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbmst\" (UniqueName: \"kubernetes.io/projected/1b2b3e72-d4ac-4407-97ae-421c5ad091a1-kube-api-access-qbmst\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7k4cr\" (UID: \"1b2b3e72-d4ac-4407-97ae-421c5ad091a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.656926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.656861 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b2b3e72-d4ac-4407-97ae-421c5ad091a1-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7k4cr\" (UID: \"1b2b3e72-d4ac-4407-97ae-421c5ad091a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.656926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.656882 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b2b3e72-d4ac-4407-97ae-421c5ad091a1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7k4cr\" (UID: \"1b2b3e72-d4ac-4407-97ae-421c5ad091a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.657293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.657256 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1b2b3e72-d4ac-4407-97ae-421c5ad091a1-home\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7k4cr\" (UID: \"1b2b3e72-d4ac-4407-97ae-421c5ad091a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.657293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.657287 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1b2b3e72-d4ac-4407-97ae-421c5ad091a1-model-cache\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7k4cr\" (UID: \"1b2b3e72-d4ac-4407-97ae-421c5ad091a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.657481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.657330 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1b2b3e72-d4ac-4407-97ae-421c5ad091a1-tmp-dir\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7k4cr\" (UID: \"1b2b3e72-d4ac-4407-97ae-421c5ad091a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.659722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.659697 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1b2b3e72-d4ac-4407-97ae-421c5ad091a1-dshm\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7k4cr\" (UID: \"1b2b3e72-d4ac-4407-97ae-421c5ad091a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.659817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.659732 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1b2b3e72-d4ac-4407-97ae-421c5ad091a1-tls-certs\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7k4cr\" (UID: \"1b2b3e72-d4ac-4407-97ae-421c5ad091a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.677575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.677538 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbmst\" (UniqueName: \"kubernetes.io/projected/1b2b3e72-d4ac-4407-97ae-421c5ad091a1-kube-api-access-qbmst\") pod \"e2e-embedding-simulated-kserve-67bf594cd6-7k4cr\" (UID: \"1b2b3e72-d4ac-4407-97ae-421c5ad091a1\") " pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:02.849734 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:02.849641 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:03.161955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:03.161808 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr"] Oct 30 18:09:03.168942 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:09:03.168899 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1b2b3e72_d4ac_4407_97ae_421c5ad091a1.slice/crio-7d8ba76f22e33cee0a058f289c7c0578918914dd446b292e47f14701eea1364a WatchSource:0}: Error finding container 7d8ba76f22e33cee0a058f289c7c0578918914dd446b292e47f14701eea1364a: Status 404 returned error can't find the container with id 7d8ba76f22e33cee0a058f289c7c0578918914dd446b292e47f14701eea1364a Oct 30 18:09:03.475743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:03.475592 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" event={"ID":"1b2b3e72-d4ac-4407-97ae-421c5ad091a1","Type":"ContainerStarted","Data":"7d8ba76f22e33cee0a058f289c7c0578918914dd446b292e47f14701eea1364a"} Oct 30 18:09:05.374185 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.374152 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2"] Oct 30 18:09:05.377544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.377520 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.394600 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.394568 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\"" Oct 30 18:09:05.407964 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.407936 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2"] Oct 30 18:09:05.484794 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.484744 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/693cdf78-86cc-4f53-9c17-3926b662aeb4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2\" (UID: \"693cdf78-86cc-4f53-9c17-3926b662aeb4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.484999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.484824 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/693cdf78-86cc-4f53-9c17-3926b662aeb4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2\" (UID: \"693cdf78-86cc-4f53-9c17-3926b662aeb4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.484999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.484946 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94lkd\" (UniqueName: \"kubernetes.io/projected/693cdf78-86cc-4f53-9c17-3926b662aeb4-kube-api-access-94lkd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2\" (UID: \"693cdf78-86cc-4f53-9c17-3926b662aeb4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.484999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.484986 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/693cdf78-86cc-4f53-9c17-3926b662aeb4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2\" (UID: \"693cdf78-86cc-4f53-9c17-3926b662aeb4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.485171 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.485045 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/693cdf78-86cc-4f53-9c17-3926b662aeb4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2\" (UID: \"693cdf78-86cc-4f53-9c17-3926b662aeb4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.485171 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.485066 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/693cdf78-86cc-4f53-9c17-3926b662aeb4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2\" (UID: \"693cdf78-86cc-4f53-9c17-3926b662aeb4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.487813 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.487784 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" event={"ID":"1b2b3e72-d4ac-4407-97ae-421c5ad091a1","Type":"ContainerStarted","Data":"28ad77678867951a97bb10bad30ca728de42733ef294ceb708c5ea9f80b182aa"} Oct 30 18:09:05.529257 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.529201 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" podStartSLOduration=2.222605403 podStartE2EDuration="3.529187839s" totalImagesPullingTime="1.306582436s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:05.529167367 +0000 UTC m=+444.845587692" watchObservedRunningTime="2026-10-30 18:09:05.529187839 +0000 UTC m=+444.845608161" Oct 30 18:09:05.586113 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.586079 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94lkd\" (UniqueName: \"kubernetes.io/projected/693cdf78-86cc-4f53-9c17-3926b662aeb4-kube-api-access-94lkd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2\" (UID: \"693cdf78-86cc-4f53-9c17-3926b662aeb4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.586352 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.586127 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/693cdf78-86cc-4f53-9c17-3926b662aeb4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2\" (UID: \"693cdf78-86cc-4f53-9c17-3926b662aeb4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.586352 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.586171 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/693cdf78-86cc-4f53-9c17-3926b662aeb4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2\" (UID: \"693cdf78-86cc-4f53-9c17-3926b662aeb4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.586352 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.586203 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/693cdf78-86cc-4f53-9c17-3926b662aeb4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2\" (UID: \"693cdf78-86cc-4f53-9c17-3926b662aeb4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.586352 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.586292 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/693cdf78-86cc-4f53-9c17-3926b662aeb4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2\" (UID: \"693cdf78-86cc-4f53-9c17-3926b662aeb4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.586352 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.586343 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/693cdf78-86cc-4f53-9c17-3926b662aeb4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2\" (UID: \"693cdf78-86cc-4f53-9c17-3926b662aeb4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.586689 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.586664 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/693cdf78-86cc-4f53-9c17-3926b662aeb4-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2\" (UID: \"693cdf78-86cc-4f53-9c17-3926b662aeb4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.586749 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.586689 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/693cdf78-86cc-4f53-9c17-3926b662aeb4-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2\" (UID: \"693cdf78-86cc-4f53-9c17-3926b662aeb4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.586800 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.586765 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/693cdf78-86cc-4f53-9c17-3926b662aeb4-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2\" (UID: \"693cdf78-86cc-4f53-9c17-3926b662aeb4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.588718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.588695 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/693cdf78-86cc-4f53-9c17-3926b662aeb4-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2\" (UID: \"693cdf78-86cc-4f53-9c17-3926b662aeb4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.588933 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.588914 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/693cdf78-86cc-4f53-9c17-3926b662aeb4-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2\" (UID: \"693cdf78-86cc-4f53-9c17-3926b662aeb4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.605654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.605622 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94lkd\" (UniqueName: \"kubernetes.io/projected/693cdf78-86cc-4f53-9c17-3926b662aeb4-kube-api-access-94lkd\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2\" (UID: \"693cdf78-86cc-4f53-9c17-3926b662aeb4\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.690771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.690686 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:05.849900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.849866 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:05.988050 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:05.988023 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2"] Oct 30 18:09:05.991653 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:09:05.991620 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod693cdf78_86cc_4f53_9c17_3926b662aeb4.slice/crio-085deea0d4dde82b3ba9cc6cc272660c5ce90de6af354cb8fa590afa98a36079 WatchSource:0}: Error finding container 085deea0d4dde82b3ba9cc6cc272660c5ce90de6af354cb8fa590afa98a36079: Status 404 returned error can't find the container with id 085deea0d4dde82b3ba9cc6cc272660c5ce90de6af354cb8fa590afa98a36079 Oct 30 18:09:06.494365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:06.494327 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" event={"ID":"693cdf78-86cc-4f53-9c17-3926b662aeb4","Type":"ContainerStarted","Data":"cce848fec46180443b12e8ac2ae49607e3013d6de27d703c0f840950b22c7600"} Oct 30 18:09:06.494842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:06.494390 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" event={"ID":"693cdf78-86cc-4f53-9c17-3926b662aeb4","Type":"ContainerStarted","Data":"085deea0d4dde82b3ba9cc6cc272660c5ce90de6af354cb8fa590afa98a36079"} Oct 30 18:09:06.534830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:06.534747 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" podStartSLOduration=1.276978802 podStartE2EDuration="1.534732531s" totalImagesPullingTime="257.753729ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:06.534066686 +0000 UTC m=+445.850487009" watchObservedRunningTime="2026-10-30 18:09:06.534732531 +0000 UTC m=+445.851152851" Oct 30 18:09:06.691803 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:06.691719 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:12.850386 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:12.850347 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:12.863041 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:12.863009 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:13.307773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.307743 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt"] Oct 30 18:09:13.311102 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.311082 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.339586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.339558 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt"] Oct 30 18:09:13.359158 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.359126 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4071cbed-2e82-4a41-97a0-ec2a30675d06-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.359308 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.359190 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071cbed-2e82-4a41-97a0-ec2a30675d06-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.359308 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.359218 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071cbed-2e82-4a41-97a0-ec2a30675d06-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.359308 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.359244 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4071cbed-2e82-4a41-97a0-ec2a30675d06-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.359308 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.359260 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjhgv\" (UniqueName: \"kubernetes.io/projected/4071cbed-2e82-4a41-97a0-ec2a30675d06-kube-api-access-hjhgv\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.359308 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.359283 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4071cbed-2e82-4a41-97a0-ec2a30675d06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.359491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.359310 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4071cbed-2e82-4a41-97a0-ec2a30675d06-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.359491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.359346 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4071cbed-2e82-4a41-97a0-ec2a30675d06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.359491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.359370 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4071cbed-2e82-4a41-97a0-ec2a30675d06-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.460647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.460619 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4071cbed-2e82-4a41-97a0-ec2a30675d06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.460844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.460656 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4071cbed-2e82-4a41-97a0-ec2a30675d06-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.460844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.460698 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4071cbed-2e82-4a41-97a0-ec2a30675d06-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.460844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.460749 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071cbed-2e82-4a41-97a0-ec2a30675d06-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.460844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.460783 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071cbed-2e82-4a41-97a0-ec2a30675d06-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.460844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.460810 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4071cbed-2e82-4a41-97a0-ec2a30675d06-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.460844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.460830 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjhgv\" (UniqueName: \"kubernetes.io/projected/4071cbed-2e82-4a41-97a0-ec2a30675d06-kube-api-access-hjhgv\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.461163 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.460861 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4071cbed-2e82-4a41-97a0-ec2a30675d06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.461163 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.460903 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4071cbed-2e82-4a41-97a0-ec2a30675d06-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.461163 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.461109 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4071cbed-2e82-4a41-97a0-ec2a30675d06-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.461354 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.461333 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071cbed-2e82-4a41-97a0-ec2a30675d06-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.461492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.461474 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4071cbed-2e82-4a41-97a0-ec2a30675d06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.461751 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.461608 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4071cbed-2e82-4a41-97a0-ec2a30675d06-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.461751 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.461720 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4071cbed-2e82-4a41-97a0-ec2a30675d06-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.463303 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.463276 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4071cbed-2e82-4a41-97a0-ec2a30675d06-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.463573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.463554 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4071cbed-2e82-4a41-97a0-ec2a30675d06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.481106 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.481083 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjhgv\" (UniqueName: \"kubernetes.io/projected/4071cbed-2e82-4a41-97a0-ec2a30675d06-kube-api-access-hjhgv\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.481333 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.481317 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4071cbed-2e82-4a41-97a0-ec2a30675d06-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-wmxgt\" (UID: \"4071cbed-2e82-4a41-97a0-ec2a30675d06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.534170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.534140 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" Oct 30 18:09:13.624289 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.624219 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:13.929907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.929809 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt"] Oct 30 18:09:13.933757 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:09:13.933733 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4071cbed_2e82_4a41_97a0_ec2a30675d06.slice/crio-f4aa3d24b665bedcef3913fa8418bfddcfe3cc963728643fd568837e5619fa18 WatchSource:0}: Error finding container f4aa3d24b665bedcef3913fa8418bfddcfe3cc963728643fd568837e5619fa18: Status 404 returned error can't find the container with id f4aa3d24b665bedcef3913fa8418bfddcfe3cc963728643fd568837e5619fa18 Oct 30 18:09:13.935919 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.935888 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:09:13.936009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.935952 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:09:13.936009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:13.935980 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:09:14.538785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:14.538748 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" event={"ID":"4071cbed-2e82-4a41-97a0-ec2a30675d06","Type":"ContainerStarted","Data":"086061adcf12383b89c3fdd1305b0aa9076231cc2142e30280bc03fca442a42a"} Oct 30 18:09:14.538991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:14.538812 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" event={"ID":"4071cbed-2e82-4a41-97a0-ec2a30675d06","Type":"ContainerStarted","Data":"f4aa3d24b665bedcef3913fa8418bfddcfe3cc963728643fd568837e5619fa18"} Oct 30 18:09:14.596753 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:14.596669 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" podStartSLOduration=1.596656699 podStartE2EDuration="1.596656699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:14.595739424 +0000 UTC m=+453.912159744" watchObservedRunningTime="2026-10-30 18:09:14.596656699 +0000 UTC m=+453.913077020" Oct 30 18:09:14.630666 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:14.630578 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:14.631944 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:14.631623 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-wmxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 18:09:14.631944 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:14.631677 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" podUID="4071cbed-2e82-4a41-97a0-ec2a30675d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 18:09:15.628921 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:15.628883 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-wmxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 18:09:15.629576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:15.629537 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" podUID="4071cbed-2e82-4a41-97a0-ec2a30675d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 18:09:15.691821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:15.691778 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:15.704631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:15.704597 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:16.558217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:16.558190 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" Oct 30 18:09:16.624936 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:16.624900 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-wmxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 18:09:16.625198 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:16.625171 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" podUID="4071cbed-2e82-4a41-97a0-ec2a30675d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 18:09:17.630930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:17.630889 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-wmxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 18:09:17.633017 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:17.632981 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" podUID="4071cbed-2e82-4a41-97a0-ec2a30675d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 18:09:18.630534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:18.629910 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-wmxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 18:09:18.630534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:18.629975 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" podUID="4071cbed-2e82-4a41-97a0-ec2a30675d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 18:09:19.634536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:19.630604 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-wmxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 18:09:19.634536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:19.630680 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" podUID="4071cbed-2e82-4a41-97a0-ec2a30675d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 18:09:20.625400 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:20.625321 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-wmxgt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" start-of-body= Oct 30 18:09:20.625400 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:20.625375 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" podUID="4071cbed-2e82-4a41-97a0-ec2a30675d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": dial tcp 10.133.0.27:15021: connect: connection refused" Oct 30 18:09:22.145049 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:22.145016 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:22.145601 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:22.145574 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:22.146214 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:22.146193 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" Oct 30 18:09:26.306282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:26.306246 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-b9778b868-g84vm"] Oct 30 18:09:26.309584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:26.309566 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b9778b868-g84vm" Oct 30 18:09:26.328069 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:26.328039 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-b9778b868-g84vm"] Oct 30 18:09:26.346359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:26.346328 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\"" Oct 30 18:09:26.347401 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:26.347345 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\"" Oct 30 18:09:26.347539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:26.347434 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-mdm7d\"" Oct 30 18:09:26.397955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:26.397918 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a3685e-2b7d-4244-96f2-db86eb3e0f7b-metrics-certs\") pod \"ai-gateway-controller-b9778b868-g84vm\" (UID: \"67a3685e-2b7d-4244-96f2-db86eb3e0f7b\") " pod="opendatahub/ai-gateway-controller-b9778b868-g84vm" Oct 30 18:09:26.397955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:26.397955 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hffdm\" (UniqueName: \"kubernetes.io/projected/67a3685e-2b7d-4244-96f2-db86eb3e0f7b-kube-api-access-hffdm\") pod \"ai-gateway-controller-b9778b868-g84vm\" (UID: \"67a3685e-2b7d-4244-96f2-db86eb3e0f7b\") " pod="opendatahub/ai-gateway-controller-b9778b868-g84vm" Oct 30 18:09:26.498935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:26.498895 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a3685e-2b7d-4244-96f2-db86eb3e0f7b-metrics-certs\") pod \"ai-gateway-controller-b9778b868-g84vm\" (UID: \"67a3685e-2b7d-4244-96f2-db86eb3e0f7b\") " pod="opendatahub/ai-gateway-controller-b9778b868-g84vm" Oct 30 18:09:26.498935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:26.498934 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hffdm\" (UniqueName: \"kubernetes.io/projected/67a3685e-2b7d-4244-96f2-db86eb3e0f7b-kube-api-access-hffdm\") pod \"ai-gateway-controller-b9778b868-g84vm\" (UID: \"67a3685e-2b7d-4244-96f2-db86eb3e0f7b\") " pod="opendatahub/ai-gateway-controller-b9778b868-g84vm" Oct 30 18:09:26.501556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:26.501533 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a3685e-2b7d-4244-96f2-db86eb3e0f7b-metrics-certs\") pod \"ai-gateway-controller-b9778b868-g84vm\" (UID: \"67a3685e-2b7d-4244-96f2-db86eb3e0f7b\") " pod="opendatahub/ai-gateway-controller-b9778b868-g84vm" Oct 30 18:09:26.518730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:26.518699 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hffdm\" (UniqueName: \"kubernetes.io/projected/67a3685e-2b7d-4244-96f2-db86eb3e0f7b-kube-api-access-hffdm\") pod \"ai-gateway-controller-b9778b868-g84vm\" (UID: \"67a3685e-2b7d-4244-96f2-db86eb3e0f7b\") " pod="opendatahub/ai-gateway-controller-b9778b868-g84vm" Oct 30 18:09:26.622229 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:26.622153 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b9778b868-g84vm" Oct 30 18:09:26.912845 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:26.912767 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-b9778b868-g84vm"] Oct 30 18:09:26.916775 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:09:26.916748 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67a3685e_2b7d_4244_96f2_db86eb3e0f7b.slice/crio-2a7c3298991600b94f60f043419d865ac2aab5970769234842613ebc1bab21e4 WatchSource:0}: Error finding container 2a7c3298991600b94f60f043419d865ac2aab5970769234842613ebc1bab21e4: Status 404 returned error can't find the container with id 2a7c3298991600b94f60f043419d865ac2aab5970769234842613ebc1bab21e4 Oct 30 18:09:27.032723 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:09:27.032688 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:09:27.601549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:27.601493 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b9778b868-g84vm" event={"ID":"67a3685e-2b7d-4244-96f2-db86eb3e0f7b","Type":"ContainerStarted","Data":"2a7c3298991600b94f60f043419d865ac2aab5970769234842613ebc1bab21e4"} Oct 30 18:09:29.610122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:29.610092 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b9778b868-g84vm" event={"ID":"67a3685e-2b7d-4244-96f2-db86eb3e0f7b","Type":"ContainerStarted","Data":"727d60dd61ed5a6a9298c206cf09814e8f2700c749ead04622bb830295d96938"} Oct 30 18:09:29.610593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:29.610220 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-b9778b868-g84vm" Oct 30 18:09:31.045812 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:31.045755 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-b9778b868-g84vm" podStartSLOduration=3.368923257 podStartE2EDuration="5.045740956s" totalImagesPullingTime="1.676817699s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:29.657196501 +0000 UTC m=+468.973616822" watchObservedRunningTime="2026-10-30 18:09:31.045740956 +0000 UTC m=+470.362161276" Oct 30 18:09:31.047095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:31.047067 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9w8hc"] Oct 30 18:09:31.050401 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:31.050381 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w8hc" Oct 30 18:09:31.078777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:31.078639 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-jbvcp\"" Oct 30 18:09:31.078777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:31.078750 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 30 18:09:31.079872 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:31.079845 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9w8hc"] Oct 30 18:09:31.143100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:31.143059 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mkmpk\" (UniqueName: \"kubernetes.io/projected/9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae-kube-api-access-mkmpk\") pod \"payload-pre-processing-7b98bb678c-9w8hc\" (UID: \"9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w8hc" Oct 30 18:09:31.143397 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:31.143373 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9w8hc\" (UID: \"9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w8hc" Oct 30 18:09:31.244896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:31.244849 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9w8hc\" (UID: \"9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w8hc" Oct 30 18:09:31.245102 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:31.244990 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mkmpk\" (UniqueName: \"kubernetes.io/projected/9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae-kube-api-access-mkmpk\") pod \"payload-pre-processing-7b98bb678c-9w8hc\" (UID: \"9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w8hc" Oct 30 18:09:31.245739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:31.245713 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae-plugins-config-volume\") pod \"payload-pre-processing-7b98bb678c-9w8hc\" (UID: \"9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w8hc" Oct 30 18:09:31.264624 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:31.264590 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mkmpk\" (UniqueName: \"kubernetes.io/projected/9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae-kube-api-access-mkmpk\") pod \"payload-pre-processing-7b98bb678c-9w8hc\" (UID: \"9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae\") " pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w8hc" Oct 30 18:09:31.362487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:31.362396 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w8hc" Oct 30 18:09:31.661621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:31.661491 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-7b98bb678c-9w8hc"] Oct 30 18:09:31.665130 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:09:31.665102 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3c5fd0_ba0e_45b0_a1b9_ac38b9dc8fae.slice/crio-3f40656844bfd3971cf00f5690d65b9a650ef6495dabc368783dbebb25cd6248 WatchSource:0}: Error finding container 3f40656844bfd3971cf00f5690d65b9a650ef6495dabc368783dbebb25cd6248: Status 404 returned error can't find the container with id 3f40656844bfd3971cf00f5690d65b9a650ef6495dabc368783dbebb25cd6248 Oct 30 18:09:32.626271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:32.626231 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w8hc" event={"ID":"9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae","Type":"ContainerStarted","Data":"3f40656844bfd3971cf00f5690d65b9a650ef6495dabc368783dbebb25cd6248"} Oct 30 18:09:34.637463 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:34.637428 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w8hc" event={"ID":"9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae","Type":"ContainerStarted","Data":"73439146a5a11e06dc8ce9af05a616ee0625b1c25f1af0e74e1d0d68dbc06119"} Oct 30 18:09:34.637960 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:34.637495 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w8hc" Oct 30 18:09:34.676710 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:34.676656 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w8hc" podStartSLOduration=1.441855813 podStartE2EDuration="3.676642762s" totalImagesPullingTime="2.234786949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:09:31 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:09:34.675992842 +0000 UTC m=+473.992413163" watchObservedRunningTime="2026-10-30 18:09:34.676642762 +0000 UTC m=+473.993063114" Oct 30 18:09:40.618176 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:40.618148 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-b9778b868-g84vm" Oct 30 18:09:45.644342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:45.644316 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-7b98bb678c-9w8hc" Oct 30 18:09:54.301363 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.301329 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-b9778b868-g84vm"] Oct 30 18:09:54.301856 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.301643 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-b9778b868-g84vm" podUID="67a3685e-2b7d-4244-96f2-db86eb3e0f7b" containerName="manager" containerID="cri-o://727d60dd61ed5a6a9298c206cf09814e8f2700c749ead04622bb830295d96938" gracePeriod=30 Oct 30 18:09:54.558273 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.558215 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b9778b868-g84vm" Oct 30 18:09:54.630160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.630130 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67a3685e-2b7d-4244-96f2-db86eb3e0f7b-kube-api-access-hffdm\" (UniqueName: \"kubernetes.io/projected/67a3685e-2b7d-4244-96f2-db86eb3e0f7b-kube-api-access-hffdm\") pod \"67a3685e-2b7d-4244-96f2-db86eb3e0f7b\" (UID: \"67a3685e-2b7d-4244-96f2-db86eb3e0f7b\") " Oct 30 18:09:54.630330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.630189 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67a3685e-2b7d-4244-96f2-db86eb3e0f7b-metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a3685e-2b7d-4244-96f2-db86eb3e0f7b-metrics-certs\") pod \"67a3685e-2b7d-4244-96f2-db86eb3e0f7b\" (UID: \"67a3685e-2b7d-4244-96f2-db86eb3e0f7b\") " Oct 30 18:09:54.632419 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.632388 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67a3685e-2b7d-4244-96f2-db86eb3e0f7b-metrics-certs" pod "67a3685e-2b7d-4244-96f2-db86eb3e0f7b" (UID: "67a3685e-2b7d-4244-96f2-db86eb3e0f7b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:09:54.632541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.632461 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67a3685e-2b7d-4244-96f2-db86eb3e0f7b-kube-api-access-hffdm" pod "67a3685e-2b7d-4244-96f2-db86eb3e0f7b" (UID: "67a3685e-2b7d-4244-96f2-db86eb3e0f7b"). InnerVolumeSpecName "kube-api-access-hffdm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:09:54.720911 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.720876 5232 generic.go:356] "Generic (PLEG): container finished" podID="67a3685e-2b7d-4244-96f2-db86eb3e0f7b" containerID="727d60dd61ed5a6a9298c206cf09814e8f2700c749ead04622bb830295d96938" exitCode=0 Oct 30 18:09:54.721075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.720921 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-b9778b868-g84vm" Oct 30 18:09:54.721075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.720952 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b9778b868-g84vm" event={"ID":"67a3685e-2b7d-4244-96f2-db86eb3e0f7b","Type":"ContainerDied","Data":"727d60dd61ed5a6a9298c206cf09814e8f2700c749ead04622bb830295d96938"} Oct 30 18:09:54.721075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.721003 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-b9778b868-g84vm" event={"ID":"67a3685e-2b7d-4244-96f2-db86eb3e0f7b","Type":"ContainerDied","Data":"2a7c3298991600b94f60f043419d865ac2aab5970769234842613ebc1bab21e4"} Oct 30 18:09:54.721075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.721024 5232 scope.go:122] "RemoveContainer" containerID="727d60dd61ed5a6a9298c206cf09814e8f2700c749ead04622bb830295d96938" Oct 30 18:09:54.729903 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.729884 5232 scope.go:122] "RemoveContainer" containerID="727d60dd61ed5a6a9298c206cf09814e8f2700c749ead04622bb830295d96938" Oct 30 18:09:54.730188 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:09:54.730171 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"727d60dd61ed5a6a9298c206cf09814e8f2700c749ead04622bb830295d96938\": container with ID starting with 727d60dd61ed5a6a9298c206cf09814e8f2700c749ead04622bb830295d96938 not found: ID does not exist" containerID="727d60dd61ed5a6a9298c206cf09814e8f2700c749ead04622bb830295d96938" Oct 30 18:09:54.730277 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.730194 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"727d60dd61ed5a6a9298c206cf09814e8f2700c749ead04622bb830295d96938"} err="failed to get container status \"727d60dd61ed5a6a9298c206cf09814e8f2700c749ead04622bb830295d96938\": rpc error: code = NotFound desc = could not find container \"727d60dd61ed5a6a9298c206cf09814e8f2700c749ead04622bb830295d96938\": container with ID starting with 727d60dd61ed5a6a9298c206cf09814e8f2700c749ead04622bb830295d96938 not found: ID does not exist" Oct 30 18:09:54.731590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.731570 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hffdm\" (UniqueName: \"kubernetes.io/projected/67a3685e-2b7d-4244-96f2-db86eb3e0f7b-kube-api-access-hffdm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:09:54.731590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.731591 5232 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/67a3685e-2b7d-4244-96f2-db86eb3e0f7b-metrics-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:09:54.771411 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.771384 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-b9778b868-g84vm"] Oct 30 18:09:54.775455 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.775435 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-b9778b868-g84vm"] Oct 30 18:09:54.800835 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:09:54.800798 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67a3685e-2b7d-4244-96f2-db86eb3e0f7b" path="/var/lib/kubelet/pods/67a3685e-2b7d-4244-96f2-db86eb3e0f7b/volumes" Oct 30 18:09:57.033132 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:09:57.033090 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:10:23.063855 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.063818 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992"] Oct 30 18:10:23.064446 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.064304 5232 cpu_manager.go:414] "containerMap: removing container" podUID="67a3685e-2b7d-4244-96f2-db86eb3e0f7b" containerName="manager" Oct 30 18:10:23.064446 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.064321 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67a3685e-2b7d-4244-96f2-db86eb3e0f7b" containerName="manager" Oct 30 18:10:23.064446 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.064432 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="67a3685e-2b7d-4244-96f2-db86eb3e0f7b" containerName="manager" Oct 30 18:10:23.067502 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.067476 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.086123 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.086101 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-fde2c5-openshift-default-dockercfg-ffxp6\"" Oct 30 18:10:23.096319 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.096294 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992"] Oct 30 18:10:23.166142 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.166111 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.166142 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.166145 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3480923d-8625-430e-819c-628e203c4946-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.166324 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.166175 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.166324 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.166246 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.166324 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.166297 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qgbjb\" (UniqueName: \"kubernetes.io/projected/3480923d-8625-430e-819c-628e203c4946-kube-api-access-qgbjb\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.166425 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.166329 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3480923d-8625-430e-819c-628e203c4946-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.166425 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.166388 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.166425 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.166405 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.166547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.166481 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3480923d-8625-430e-819c-628e203c4946-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.183595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.183568 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974"] Oct 30 18:10:23.186835 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.186814 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.202813 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.202790 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-6ce5eb-openshift-default-dockercfg-td7nf\"" Oct 30 18:10:23.216572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.216544 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974"] Oct 30 18:10:23.267787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.267758 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.267936 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.267793 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhjxm\" (UniqueName: \"kubernetes.io/projected/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-kube-api-access-qhjxm\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.267936 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.267896 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.268024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.267938 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.268024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.267977 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.268024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.268008 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3480923d-8625-430e-819c-628e203c4946-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.268135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.268067 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.268135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.268109 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.268135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.268128 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.268260 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.268152 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qgbjb\" (UniqueName: \"kubernetes.io/projected/3480923d-8625-430e-819c-628e203c4946-kube-api-access-qgbjb\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.268260 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.268184 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.268260 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.268206 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3480923d-8625-430e-819c-628e203c4946-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.268260 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.268243 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.268466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.268272 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.268466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.268294 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.268466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.268350 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.268466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.268414 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.268466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.268453 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.268690 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.268464 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.268690 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.268546 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3480923d-8625-430e-819c-628e203c4946-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.268690 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.268620 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.268690 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.268674 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.269049 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.269033 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3480923d-8625-430e-819c-628e203c4946-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.270369 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.270339 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.270495 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.270479 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3480923d-8625-430e-819c-628e203c4946-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.290678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.290651 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qgbjb\" (UniqueName: \"kubernetes.io/projected/3480923d-8625-430e-819c-628e203c4946-kube-api-access-qgbjb\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.293397 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.293378 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3480923d-8625-430e-819c-628e203c4946-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.369347 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.369260 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.369347 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.369314 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.369606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.369436 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.369606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.369475 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.369606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.369543 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.369606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.369567 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhjxm\" (UniqueName: \"kubernetes.io/projected/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-kube-api-access-qhjxm\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.369817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.369620 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.369817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.369674 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.369817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.369722 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.369817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.369783 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.370076 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.369956 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.370076 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.370013 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.370142 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.370094 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.370742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.370720 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.371996 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.371971 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.372301 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.372283 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.380165 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.380146 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:23.389349 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.389323 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.390359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.390337 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhjxm\" (UniqueName: \"kubernetes.io/projected/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-kube-api-access-qhjxm\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.501104 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.501071 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:23.575653 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.575625 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx"] Oct 30 18:10:23.580130 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.580103 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.605958 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.605932 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx"] Oct 30 18:10:23.626266 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.626181 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-ef13cc-openshift-default-dockercfg-5k66q\"" Oct 30 18:10:23.672634 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.672606 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.672634 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.672637 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.672829 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.672670 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.672872 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.672827 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.672914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.672895 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.672952 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.672929 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntbss\" (UniqueName: \"kubernetes.io/projected/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-kube-api-access-ntbss\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.673022 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.673010 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.673059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.673042 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.673100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.673060 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.693859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.693830 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992"] Oct 30 18:10:23.698151 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:10:23.698126 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3480923d_8625_430e_819c_628e203c4946.slice/crio-c355715b5eb8d4b875320be8ab53a1c8d014e055b22210a66c0feb7c95da26ff WatchSource:0}: Error finding container c355715b5eb8d4b875320be8ab53a1c8d014e055b22210a66c0feb7c95da26ff: Status 404 returned error can't find the container with id c355715b5eb8d4b875320be8ab53a1c8d014e055b22210a66c0feb7c95da26ff Oct 30 18:10:23.700436 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.700394 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:10:23.700558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.700485 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:10:23.700558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.700552 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:10:23.774096 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.774061 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.774248 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.774115 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntbss\" (UniqueName: \"kubernetes.io/projected/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-kube-api-access-ntbss\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.774248 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.774160 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.774402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.774340 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.774402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.774394 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.774550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.774466 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.774550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.774501 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.774748 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.774722 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.774868 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.774759 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.774868 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.774808 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.774868 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.774857 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.775008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.774929 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.775540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.775442 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.775958 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.775938 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.777059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.777036 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.777766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.777750 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.804498 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.804441 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntbss\" (UniqueName: \"kubernetes.io/projected/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-kube-api-access-ntbss\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.807795 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.807767 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:23.808916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.808893 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974"] Oct 30 18:10:23.813474 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:10:23.813445 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8c0e348_2b8d_48e2_9ec0_a21a83db391e.slice/crio-32e89529d4fea30aabe02d482886c5910273e88d428c15413b6bf412eeb66942 WatchSource:0}: Error finding container 32e89529d4fea30aabe02d482886c5910273e88d428c15413b6bf412eeb66942: Status 404 returned error can't find the container with id 32e89529d4fea30aabe02d482886c5910273e88d428c15413b6bf412eeb66942 Oct 30 18:10:23.815924 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.815886 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:10:23.816014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.815969 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:10:23.816070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.816012 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:10:23.845113 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.845085 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" event={"ID":"f8c0e348-2b8d-48e2-9ec0-a21a83db391e","Type":"ContainerStarted","Data":"32e89529d4fea30aabe02d482886c5910273e88d428c15413b6bf412eeb66942"} Oct 30 18:10:23.846942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.846913 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" event={"ID":"3480923d-8625-430e-819c-628e203c4946","Type":"ContainerStarted","Data":"2b789eacda30691b99761c368abbc2e45d23be596696dfda30a2a6e4ce8b8202"} Oct 30 18:10:23.847047 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.846968 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" event={"ID":"3480923d-8625-430e-819c-628e203c4946","Type":"ContainerStarted","Data":"c355715b5eb8d4b875320be8ab53a1c8d014e055b22210a66c0feb7c95da26ff"} Oct 30 18:10:23.898672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.896019 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" podStartSLOduration=0.895999951 podStartE2EDuration="895.999951ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:23.889176869 +0000 UTC m=+523.205597193" watchObservedRunningTime="2026-10-30 18:10:23.895999951 +0000 UTC m=+523.212420273" Oct 30 18:10:23.898672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:23.898360 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:24.206666 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:24.206588 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx"] Oct 30 18:10:24.210555 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:10:24.210525 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podffe6b1f9_81fc_4d91_831e_7ddcb0668a1e.slice/crio-2a2b18286f1768f929a4e75002ae53aa83d0bb56061f0d30712fca9fcdc9aa72 WatchSource:0}: Error finding container 2a2b18286f1768f929a4e75002ae53aa83d0bb56061f0d30712fca9fcdc9aa72: Status 404 returned error can't find the container with id 2a2b18286f1768f929a4e75002ae53aa83d0bb56061f0d30712fca9fcdc9aa72 Oct 30 18:10:24.212911 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:24.212874 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:10:24.213009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:24.212956 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:10:24.213009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:24.212988 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:10:24.381336 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:24.381299 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:24.386547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:24.386492 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:24.852171 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:24.852125 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" event={"ID":"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e","Type":"ContainerStarted","Data":"18d030245395eb9dfb629d43f057cbbfd732280822d9d92f1debf184fe1e8296"} Oct 30 18:10:24.852376 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:24.852190 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" event={"ID":"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e","Type":"ContainerStarted","Data":"2a2b18286f1768f929a4e75002ae53aa83d0bb56061f0d30712fca9fcdc9aa72"} Oct 30 18:10:24.853781 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:24.853752 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" event={"ID":"f8c0e348-2b8d-48e2-9ec0-a21a83db391e","Type":"ContainerStarted","Data":"c644ecb172cd9640e48f6b6b1a0f8c1dc380318e906d876cce36c4fecfb5055b"} Oct 30 18:10:24.854047 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:24.854020 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:24.854871 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:24.854854 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:24.901547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:24.901451 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:24.925375 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:24.925325 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" podStartSLOduration=1.9253100490000001 podStartE2EDuration="1.925310049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:24.923096965 +0000 UTC m=+524.239517288" watchObservedRunningTime="2026-10-30 18:10:24.925310049 +0000 UTC m=+524.241730414" Oct 30 18:10:25.010247 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:25.010179 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" podStartSLOduration=2.01016353 podStartE2EDuration="2.01016353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:25.009171636 +0000 UTC m=+524.325591957" watchObservedRunningTime="2026-10-30 18:10:25.01016353 +0000 UTC m=+524.326583909" Oct 30 18:10:25.501423 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:25.501382 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:25.506298 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:25.506277 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:25.880013 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:25.879927 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:25.881129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:25.881102 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:10:25.904544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:25.904491 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:26.888384 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:26.888354 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:26.889069 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:26.889053 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:10:27.033685 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:10:27.033649 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:10:29.137378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.137346 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl"] Oct 30 18:10:29.140931 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.140912 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" Oct 30 18:10:29.158393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.158367 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-fde2c5\"" Oct 30 18:10:29.171036 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.171014 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-fde2c5-dockercfg-kxglj\"" Oct 30 18:10:29.172582 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.172559 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl"] Oct 30 18:10:29.227427 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.227392 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7"] Oct 30 18:10:29.232548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.231654 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" Oct 30 18:10:29.250529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.235686 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8p9lp\" (UniqueName: \"kubernetes.io/projected/1f3f54d2-9382-4abd-81a5-4fba85c28dad-kube-api-access-8p9lp\") pod \"payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl\" (UID: \"1f3f54d2-9382-4abd-81a5-4fba85c28dad\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" Oct 30 18:10:29.250529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.235789 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3f54d2-9382-4abd-81a5-4fba85c28dad-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl\" (UID: \"1f3f54d2-9382-4abd-81a5-4fba85c28dad\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" Oct 30 18:10:29.250529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.248608 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7"] Oct 30 18:10:29.336573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.336539 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8p9lp\" (UniqueName: \"kubernetes.io/projected/1f3f54d2-9382-4abd-81a5-4fba85c28dad-kube-api-access-8p9lp\") pod \"payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl\" (UID: \"1f3f54d2-9382-4abd-81a5-4fba85c28dad\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" Oct 30 18:10:29.336763 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.336595 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048571d7-450a-40b4-8c18-94ec10221b86-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7\" (UID: \"048571d7-450a-40b4-8c18-94ec10221b86\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" Oct 30 18:10:29.336763 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.336619 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6dd4\" (UniqueName: \"kubernetes.io/projected/048571d7-450a-40b4-8c18-94ec10221b86-kube-api-access-w6dd4\") pod \"payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7\" (UID: \"048571d7-450a-40b4-8c18-94ec10221b86\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" Oct 30 18:10:29.336763 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.336658 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3f54d2-9382-4abd-81a5-4fba85c28dad-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl\" (UID: \"1f3f54d2-9382-4abd-81a5-4fba85c28dad\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" Oct 30 18:10:29.337346 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.337318 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3f54d2-9382-4abd-81a5-4fba85c28dad-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl\" (UID: \"1f3f54d2-9382-4abd-81a5-4fba85c28dad\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" Oct 30 18:10:29.356038 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.356008 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8p9lp\" (UniqueName: \"kubernetes.io/projected/1f3f54d2-9382-4abd-81a5-4fba85c28dad-kube-api-access-8p9lp\") pod \"payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl\" (UID: \"1f3f54d2-9382-4abd-81a5-4fba85c28dad\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" Oct 30 18:10:29.437651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.437567 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048571d7-450a-40b4-8c18-94ec10221b86-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7\" (UID: \"048571d7-450a-40b4-8c18-94ec10221b86\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" Oct 30 18:10:29.437651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.437609 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dd4\" (UniqueName: \"kubernetes.io/projected/048571d7-450a-40b4-8c18-94ec10221b86-kube-api-access-w6dd4\") pod \"payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7\" (UID: \"048571d7-450a-40b4-8c18-94ec10221b86\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" Oct 30 18:10:29.438242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.438220 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048571d7-450a-40b4-8c18-94ec10221b86-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7\" (UID: \"048571d7-450a-40b4-8c18-94ec10221b86\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" Oct 30 18:10:29.454154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.454130 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" Oct 30 18:10:29.457802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.457777 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6dd4\" (UniqueName: \"kubernetes.io/projected/048571d7-450a-40b4-8c18-94ec10221b86-kube-api-access-w6dd4\") pod \"payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7\" (UID: \"048571d7-450a-40b4-8c18-94ec10221b86\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" Oct 30 18:10:29.574791 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.574765 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" Oct 30 18:10:29.750632 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.750598 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl"] Oct 30 18:10:29.753868 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:10:29.753839 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1f3f54d2_9382_4abd_81a5_4fba85c28dad.slice/crio-22e2954ac93f5cd2cc2863badb7f31cfa269163903a84e1153c595b5003aebee WatchSource:0}: Error finding container 22e2954ac93f5cd2cc2863badb7f31cfa269163903a84e1153c595b5003aebee: Status 404 returned error can't find the container with id 22e2954ac93f5cd2cc2863badb7f31cfa269163903a84e1153c595b5003aebee Oct 30 18:10:29.869276 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.869244 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7"] Oct 30 18:10:29.873100 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:10:29.873070 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod048571d7_450a_40b4_8c18_94ec10221b86.slice/crio-cc896570eb90dfc3b2245d7b5d0b1439c4392d69dda0c85c208e749ce7eaec08 WatchSource:0}: Error finding container cc896570eb90dfc3b2245d7b5d0b1439c4392d69dda0c85c208e749ce7eaec08: Status 404 returned error can't find the container with id cc896570eb90dfc3b2245d7b5d0b1439c4392d69dda0c85c208e749ce7eaec08 Oct 30 18:10:29.903098 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.903068 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" event={"ID":"1f3f54d2-9382-4abd-81a5-4fba85c28dad","Type":"ContainerStarted","Data":"b6f8d0ddbb6dad0de774503616b83ab4ec48d5c7c6d3c37dcf9a0d34f7558eea"} Oct 30 18:10:29.903224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.903130 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" event={"ID":"1f3f54d2-9382-4abd-81a5-4fba85c28dad","Type":"ContainerStarted","Data":"22e2954ac93f5cd2cc2863badb7f31cfa269163903a84e1153c595b5003aebee"} Oct 30 18:10:29.903224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.903163 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" Oct 30 18:10:29.904363 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.904340 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" event={"ID":"048571d7-450a-40b4-8c18-94ec10221b86","Type":"ContainerStarted","Data":"cc896570eb90dfc3b2245d7b5d0b1439c4392d69dda0c85c208e749ce7eaec08"} Oct 30 18:10:29.944836 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:29.944769 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" podStartSLOduration=0.944749331 podStartE2EDuration="944.749331ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:29.943085082 +0000 UTC m=+529.259505406" watchObservedRunningTime="2026-10-30 18:10:29.944749331 +0000 UTC m=+529.261169648" Oct 30 18:10:30.911980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:30.911947 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" event={"ID":"048571d7-450a-40b4-8c18-94ec10221b86","Type":"ContainerStarted","Data":"2321ac6555ed9e7b126d2a02fc840331f148df60159a950adb5261d090c50ca8"} Oct 30 18:10:30.912487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:30.912036 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" Oct 30 18:10:30.955414 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:30.955358 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" podStartSLOduration=1.9553421640000002 podStartE2EDuration="1.955342164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:30.952408047 +0000 UTC m=+530.268828369" watchObservedRunningTime="2026-10-30 18:10:30.955342164 +0000 UTC m=+530.271762485" Oct 30 18:10:31.067983 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.067952 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj"] Oct 30 18:10:31.071525 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.071482 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" Oct 30 18:10:31.092397 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.092360 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-fb2fa0-dockercfg-djfxp\"" Oct 30 18:10:31.092551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.092480 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-fb2fa0\"" Oct 30 18:10:31.094614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.094586 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj"] Oct 30 18:10:31.155041 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.155007 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d"] Oct 30 18:10:31.173600 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.167897 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" Oct 30 18:10:31.173600 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.168349 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9433bec1-83df-429a-98a3-4b573d4fb5f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj\" (UID: \"9433bec1-83df-429a-98a3-4b573d4fb5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" Oct 30 18:10:31.173600 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.168600 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8r54\" (UniqueName: \"kubernetes.io/projected/9433bec1-83df-429a-98a3-4b573d4fb5f6-kube-api-access-h8r54\") pod \"payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj\" (UID: \"9433bec1-83df-429a-98a3-4b573d4fb5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" Oct 30 18:10:31.184533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.183926 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d"] Oct 30 18:10:31.269728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.269698 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nt68\" (UniqueName: \"kubernetes.io/projected/fc97fed8-9a76-4696-ab34-5a4a1d401ac3-kube-api-access-6nt68\") pod \"payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d\" (UID: \"fc97fed8-9a76-4696-ab34-5a4a1d401ac3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" Oct 30 18:10:31.269919 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.269748 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc97fed8-9a76-4696-ab34-5a4a1d401ac3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d\" (UID: \"fc97fed8-9a76-4696-ab34-5a4a1d401ac3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" Oct 30 18:10:31.269919 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.269778 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9433bec1-83df-429a-98a3-4b573d4fb5f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj\" (UID: \"9433bec1-83df-429a-98a3-4b573d4fb5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" Oct 30 18:10:31.269919 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.269883 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8r54\" (UniqueName: \"kubernetes.io/projected/9433bec1-83df-429a-98a3-4b573d4fb5f6-kube-api-access-h8r54\") pod \"payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj\" (UID: \"9433bec1-83df-429a-98a3-4b573d4fb5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" Oct 30 18:10:31.270415 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.270399 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9433bec1-83df-429a-98a3-4b573d4fb5f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj\" (UID: \"9433bec1-83df-429a-98a3-4b573d4fb5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" Oct 30 18:10:31.289964 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.289931 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8r54\" (UniqueName: \"kubernetes.io/projected/9433bec1-83df-429a-98a3-4b573d4fb5f6-kube-api-access-h8r54\") pod \"payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj\" (UID: \"9433bec1-83df-429a-98a3-4b573d4fb5f6\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" Oct 30 18:10:31.370915 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.370878 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nt68\" (UniqueName: \"kubernetes.io/projected/fc97fed8-9a76-4696-ab34-5a4a1d401ac3-kube-api-access-6nt68\") pod \"payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d\" (UID: \"fc97fed8-9a76-4696-ab34-5a4a1d401ac3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" Oct 30 18:10:31.371071 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.370932 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc97fed8-9a76-4696-ab34-5a4a1d401ac3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d\" (UID: \"fc97fed8-9a76-4696-ab34-5a4a1d401ac3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" Oct 30 18:10:31.371621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.371604 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc97fed8-9a76-4696-ab34-5a4a1d401ac3-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d\" (UID: \"fc97fed8-9a76-4696-ab34-5a4a1d401ac3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" Oct 30 18:10:31.384830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.384802 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" Oct 30 18:10:31.390080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.390056 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nt68\" (UniqueName: \"kubernetes.io/projected/fc97fed8-9a76-4696-ab34-5a4a1d401ac3-kube-api-access-6nt68\") pod \"payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d\" (UID: \"fc97fed8-9a76-4696-ab34-5a4a1d401ac3\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" Oct 30 18:10:31.498779 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.498745 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" Oct 30 18:10:31.685559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.685524 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj"] Oct 30 18:10:31.689971 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:10:31.689924 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9433bec1_83df_429a_98a3_4b573d4fb5f6.slice/crio-c27595961475d25535d77368e1788bd45978ccba68a8f888bf9ef3329a4fa146 WatchSource:0}: Error finding container c27595961475d25535d77368e1788bd45978ccba68a8f888bf9ef3329a4fa146: Status 404 returned error can't find the container with id c27595961475d25535d77368e1788bd45978ccba68a8f888bf9ef3329a4fa146 Oct 30 18:10:31.799918 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.799893 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d"] Oct 30 18:10:31.803660 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:10:31.803627 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc97fed8_9a76_4696_ab34_5a4a1d401ac3.slice/crio-b58520dc75782a98a90090241314659687cfc7ff16a75047d229edc2f554a26c WatchSource:0}: Error finding container b58520dc75782a98a90090241314659687cfc7ff16a75047d229edc2f554a26c: Status 404 returned error can't find the container with id b58520dc75782a98a90090241314659687cfc7ff16a75047d229edc2f554a26c Oct 30 18:10:31.918644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.918612 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" event={"ID":"fc97fed8-9a76-4696-ab34-5a4a1d401ac3","Type":"ContainerStarted","Data":"4a9a7043a7f3301f1aea78e41a0f5f2d48f5deb104e8c7651932d6ed2e319fff"} Oct 30 18:10:31.919092 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.918666 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" event={"ID":"fc97fed8-9a76-4696-ab34-5a4a1d401ac3","Type":"ContainerStarted","Data":"b58520dc75782a98a90090241314659687cfc7ff16a75047d229edc2f554a26c"} Oct 30 18:10:31.919092 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.918819 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" Oct 30 18:10:31.920305 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.920280 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" event={"ID":"9433bec1-83df-429a-98a3-4b573d4fb5f6","Type":"ContainerStarted","Data":"c03ea0f6812a8a607cf8f7f10635cfc832de40a11e0d0d31c6d2fd51817355fd"} Oct 30 18:10:31.920428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.920318 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" event={"ID":"9433bec1-83df-429a-98a3-4b573d4fb5f6","Type":"ContainerStarted","Data":"c27595961475d25535d77368e1788bd45978ccba68a8f888bf9ef3329a4fa146"} Oct 30 18:10:31.920487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.920428 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" Oct 30 18:10:31.960392 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:31.960295 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" podStartSLOduration=0.960281437 podStartE2EDuration="960.281437ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:31.958719653 +0000 UTC m=+531.275139974" watchObservedRunningTime="2026-10-30 18:10:31.960281437 +0000 UTC m=+531.276701757" Oct 30 18:10:32.001138 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.001097 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" podStartSLOduration=1.001084202 podStartE2EDuration="1.001084202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:32.000154935 +0000 UTC m=+531.316575283" watchObservedRunningTime="2026-10-30 18:10:32.001084202 +0000 UTC m=+531.317504553" Oct 30 18:10:32.113619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.113586 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9"] Oct 30 18:10:32.116737 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.116713 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" Oct 30 18:10:32.133461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.133433 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-fdf4da\"" Oct 30 18:10:32.134372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.134186 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-fdf4da-dockercfg-6k58s\"" Oct 30 18:10:32.136184 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.136161 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9"] Oct 30 18:10:32.183530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.183080 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964b7eb3-281f-4006-817e-6cb694f24dac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9\" (UID: \"964b7eb3-281f-4006-817e-6cb694f24dac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" Oct 30 18:10:32.183530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.183147 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h4hjz\" (UniqueName: \"kubernetes.io/projected/964b7eb3-281f-4006-817e-6cb694f24dac-kube-api-access-h4hjz\") pod \"payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9\" (UID: \"964b7eb3-281f-4006-817e-6cb694f24dac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" Oct 30 18:10:32.232998 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.229815 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b"] Oct 30 18:10:32.235436 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.235411 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" Oct 30 18:10:32.248108 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.248079 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b"] Oct 30 18:10:32.284780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.284728 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964b7eb3-281f-4006-817e-6cb694f24dac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9\" (UID: \"964b7eb3-281f-4006-817e-6cb694f24dac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" Oct 30 18:10:32.284780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.284775 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h4hjz\" (UniqueName: \"kubernetes.io/projected/964b7eb3-281f-4006-817e-6cb694f24dac-kube-api-access-h4hjz\") pod \"payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9\" (UID: \"964b7eb3-281f-4006-817e-6cb694f24dac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" Oct 30 18:10:32.285716 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.285687 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964b7eb3-281f-4006-817e-6cb694f24dac-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9\" (UID: \"964b7eb3-281f-4006-817e-6cb694f24dac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" Oct 30 18:10:32.305712 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.305687 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h4hjz\" (UniqueName: \"kubernetes.io/projected/964b7eb3-281f-4006-817e-6cb694f24dac-kube-api-access-h4hjz\") pod \"payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9\" (UID: \"964b7eb3-281f-4006-817e-6cb694f24dac\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" Oct 30 18:10:32.386077 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.386020 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86tqt\" (UniqueName: \"kubernetes.io/projected/158c3671-9513-4f20-bdca-2801ec139c14-kube-api-access-86tqt\") pod \"payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b\" (UID: \"158c3671-9513-4f20-bdca-2801ec139c14\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" Oct 30 18:10:32.386283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.386163 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158c3671-9513-4f20-bdca-2801ec139c14-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b\" (UID: \"158c3671-9513-4f20-bdca-2801ec139c14\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" Oct 30 18:10:32.429637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.429603 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" Oct 30 18:10:32.487275 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.487214 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86tqt\" (UniqueName: \"kubernetes.io/projected/158c3671-9513-4f20-bdca-2801ec139c14-kube-api-access-86tqt\") pod \"payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b\" (UID: \"158c3671-9513-4f20-bdca-2801ec139c14\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" Oct 30 18:10:32.487380 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.487314 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158c3671-9513-4f20-bdca-2801ec139c14-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b\" (UID: \"158c3671-9513-4f20-bdca-2801ec139c14\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" Oct 30 18:10:32.488343 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.488319 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158c3671-9513-4f20-bdca-2801ec139c14-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b\" (UID: \"158c3671-9513-4f20-bdca-2801ec139c14\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" Oct 30 18:10:32.508649 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.508587 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86tqt\" (UniqueName: \"kubernetes.io/projected/158c3671-9513-4f20-bdca-2801ec139c14-kube-api-access-86tqt\") pod \"payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b\" (UID: \"158c3671-9513-4f20-bdca-2801ec139c14\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" Oct 30 18:10:32.551486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.551460 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" Oct 30 18:10:32.740438 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.740357 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9"] Oct 30 18:10:32.744999 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:10:32.744972 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod964b7eb3_281f_4006_817e_6cb694f24dac.slice/crio-9739268a2b1709c76931a8314191315e3369463d981b51b364784f343d99f9ee WatchSource:0}: Error finding container 9739268a2b1709c76931a8314191315e3369463d981b51b364784f343d99f9ee: Status 404 returned error can't find the container with id 9739268a2b1709c76931a8314191315e3369463d981b51b364784f343d99f9ee Oct 30 18:10:32.850174 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.850138 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b"] Oct 30 18:10:32.854042 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:10:32.854017 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod158c3671_9513_4f20_bdca_2801ec139c14.slice/crio-845d27504d5035330d4bb169942f111dac30dccb22f5f3dcf8f1dc7385a53f11 WatchSource:0}: Error finding container 845d27504d5035330d4bb169942f111dac30dccb22f5f3dcf8f1dc7385a53f11: Status 404 returned error can't find the container with id 845d27504d5035330d4bb169942f111dac30dccb22f5f3dcf8f1dc7385a53f11 Oct 30 18:10:32.927095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.927057 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" event={"ID":"964b7eb3-281f-4006-817e-6cb694f24dac","Type":"ContainerStarted","Data":"6f6ac0aa1ce4aed817bc78764f91776383b136572ea5987028f524f6f324c2d4"} Oct 30 18:10:32.927488 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.927200 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" Oct 30 18:10:32.927488 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.927308 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" event={"ID":"964b7eb3-281f-4006-817e-6cb694f24dac","Type":"ContainerStarted","Data":"9739268a2b1709c76931a8314191315e3369463d981b51b364784f343d99f9ee"} Oct 30 18:10:32.928924 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.928896 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" event={"ID":"158c3671-9513-4f20-bdca-2801ec139c14","Type":"ContainerStarted","Data":"1bb119c347160d9a63c813acb5d09ec9ff19ac59fb9624e8779d42973c717d44"} Oct 30 18:10:32.929026 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.928948 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" event={"ID":"158c3671-9513-4f20-bdca-2801ec139c14","Type":"ContainerStarted","Data":"845d27504d5035330d4bb169942f111dac30dccb22f5f3dcf8f1dc7385a53f11"} Oct 30 18:10:32.929026 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.928995 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" Oct 30 18:10:32.968771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:32.968722 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" podStartSLOduration=0.968708705 podStartE2EDuration="968.708705ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:32.968040182 +0000 UTC m=+532.284460505" watchObservedRunningTime="2026-10-30 18:10:32.968708705 +0000 UTC m=+532.285129025" Oct 30 18:10:33.008441 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:33.008356 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" podStartSLOduration=1.008342784 podStartE2EDuration="1.008342784s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:33.007428083 +0000 UTC m=+532.323848404" watchObservedRunningTime="2026-10-30 18:10:33.008342784 +0000 UTC m=+532.324763105" Oct 30 18:10:33.182533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:33.174438 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6"] Oct 30 18:10:33.187868 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:33.187836 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" Oct 30 18:10:33.194210 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:33.194165 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6"] Oct 30 18:10:33.202636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:33.202602 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w3-8b6cf1\"" Oct 30 18:10:33.202902 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:33.202878 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w3-8b6cf1-dockercfg-mmwdg\"" Oct 30 18:10:33.305870 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:33.305784 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nnlcc\" (UniqueName: \"kubernetes.io/projected/47a6c676-1f34-4d08-81e9-80051001aa80-kube-api-access-nnlcc\") pod \"payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6\" (UID: \"47a6c676-1f34-4d08-81e9-80051001aa80\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" Oct 30 18:10:33.305870 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:33.305853 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a6c676-1f34-4d08-81e9-80051001aa80-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6\" (UID: \"47a6c676-1f34-4d08-81e9-80051001aa80\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" Oct 30 18:10:33.406811 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:33.406774 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a6c676-1f34-4d08-81e9-80051001aa80-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6\" (UID: \"47a6c676-1f34-4d08-81e9-80051001aa80\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" Oct 30 18:10:33.406986 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:33.406928 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nnlcc\" (UniqueName: \"kubernetes.io/projected/47a6c676-1f34-4d08-81e9-80051001aa80-kube-api-access-nnlcc\") pod \"payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6\" (UID: \"47a6c676-1f34-4d08-81e9-80051001aa80\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" Oct 30 18:10:33.407556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:33.407536 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a6c676-1f34-4d08-81e9-80051001aa80-plugins-config-volume\") pod \"payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6\" (UID: \"47a6c676-1f34-4d08-81e9-80051001aa80\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" Oct 30 18:10:33.430263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:33.430232 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nnlcc\" (UniqueName: \"kubernetes.io/projected/47a6c676-1f34-4d08-81e9-80051001aa80-kube-api-access-nnlcc\") pod \"payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6\" (UID: \"47a6c676-1f34-4d08-81e9-80051001aa80\") " pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" Oct 30 18:10:33.510483 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:33.510447 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" Oct 30 18:10:33.806467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:33.806435 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6"] Oct 30 18:10:33.809788 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:10:33.809761 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47a6c676_1f34_4d08_81e9_80051001aa80.slice/crio-04cb40595ba7d67d12ee0c11d94440d7cec0be8335aecdccdeb252fba6f1c7fa WatchSource:0}: Error finding container 04cb40595ba7d67d12ee0c11d94440d7cec0be8335aecdccdeb252fba6f1c7fa: Status 404 returned error can't find the container with id 04cb40595ba7d67d12ee0c11d94440d7cec0be8335aecdccdeb252fba6f1c7fa Oct 30 18:10:33.956233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:33.956142 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" event={"ID":"47a6c676-1f34-4d08-81e9-80051001aa80","Type":"ContainerStarted","Data":"3aebff89f6fffffa889ccfff11a77b394bedcb71289ba904029454a339bf8385"} Oct 30 18:10:33.956233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:33.956205 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" event={"ID":"47a6c676-1f34-4d08-81e9-80051001aa80","Type":"ContainerStarted","Data":"04cb40595ba7d67d12ee0c11d94440d7cec0be8335aecdccdeb252fba6f1c7fa"} Oct 30 18:10:33.996860 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:33.996808 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" podStartSLOduration=0.996792244 podStartE2EDuration="996.792244ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:33.995083516 +0000 UTC m=+533.311503838" watchObservedRunningTime="2026-10-30 18:10:33.996792244 +0000 UTC m=+533.313212924" Oct 30 18:10:34.968355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:34.968320 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" Oct 30 18:10:36.787824 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:36.787790 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z"] Oct 30 18:10:36.791558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:36.791534 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" Oct 30 18:10:36.809233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:36.809203 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-ef13cc\"" Oct 30 18:10:36.821177 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:36.821150 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-ef13cc-dockercfg-fw28x\"" Oct 30 18:10:36.822816 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:36.822789 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z"] Oct 30 18:10:36.865632 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:36.865604 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m"] Oct 30 18:10:36.869311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:36.869293 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" Oct 30 18:10:36.900556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:36.900500 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m"] Oct 30 18:10:36.948839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:36.948790 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xktzg\" (UniqueName: \"kubernetes.io/projected/552542b1-fdd2-417f-88a5-614f087e6228-kube-api-access-xktzg\") pod \"payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m\" (UID: \"552542b1-fdd2-417f-88a5-614f087e6228\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" Oct 30 18:10:36.949170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:36.949146 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/771db49d-214d-4901-8885-50e40d3d4d88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z\" (UID: \"771db49d-214d-4901-8885-50e40d3d4d88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" Oct 30 18:10:36.949481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:36.949461 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwztx\" (UniqueName: \"kubernetes.io/projected/771db49d-214d-4901-8885-50e40d3d4d88-kube-api-access-gwztx\") pod \"payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z\" (UID: \"771db49d-214d-4901-8885-50e40d3d4d88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" Oct 30 18:10:36.949705 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:36.949650 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552542b1-fdd2-417f-88a5-614f087e6228-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m\" (UID: \"552542b1-fdd2-417f-88a5-614f087e6228\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" Oct 30 18:10:37.050746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:37.050668 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwztx\" (UniqueName: \"kubernetes.io/projected/771db49d-214d-4901-8885-50e40d3d4d88-kube-api-access-gwztx\") pod \"payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z\" (UID: \"771db49d-214d-4901-8885-50e40d3d4d88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" Oct 30 18:10:37.050746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:37.050707 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552542b1-fdd2-417f-88a5-614f087e6228-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m\" (UID: \"552542b1-fdd2-417f-88a5-614f087e6228\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" Oct 30 18:10:37.050746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:37.050749 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xktzg\" (UniqueName: \"kubernetes.io/projected/552542b1-fdd2-417f-88a5-614f087e6228-kube-api-access-xktzg\") pod \"payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m\" (UID: \"552542b1-fdd2-417f-88a5-614f087e6228\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" Oct 30 18:10:37.051039 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:37.050775 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/771db49d-214d-4901-8885-50e40d3d4d88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z\" (UID: \"771db49d-214d-4901-8885-50e40d3d4d88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" Oct 30 18:10:37.051498 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:37.051474 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552542b1-fdd2-417f-88a5-614f087e6228-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m\" (UID: \"552542b1-fdd2-417f-88a5-614f087e6228\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" Oct 30 18:10:37.051577 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:37.051527 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/771db49d-214d-4901-8885-50e40d3d4d88-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z\" (UID: \"771db49d-214d-4901-8885-50e40d3d4d88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" Oct 30 18:10:37.070865 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:37.070839 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xktzg\" (UniqueName: \"kubernetes.io/projected/552542b1-fdd2-417f-88a5-614f087e6228-kube-api-access-xktzg\") pod \"payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m\" (UID: \"552542b1-fdd2-417f-88a5-614f087e6228\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" Oct 30 18:10:37.070981 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:37.070901 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwztx\" (UniqueName: \"kubernetes.io/projected/771db49d-214d-4901-8885-50e40d3d4d88-kube-api-access-gwztx\") pod \"payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z\" (UID: \"771db49d-214d-4901-8885-50e40d3d4d88\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" Oct 30 18:10:37.106434 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:37.106376 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" Oct 30 18:10:37.202590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:37.202566 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" Oct 30 18:10:37.452819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:37.452743 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z"] Oct 30 18:10:37.456716 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:10:37.456678 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod771db49d_214d_4901_8885_50e40d3d4d88.slice/crio-52d9932d4a14eeb2f7fdf1018c180f3191ad25581c674a984a0a484b578f6af6 WatchSource:0}: Error finding container 52d9932d4a14eeb2f7fdf1018c180f3191ad25581c674a984a0a484b578f6af6: Status 404 returned error can't find the container with id 52d9932d4a14eeb2f7fdf1018c180f3191ad25581c674a984a0a484b578f6af6 Oct 30 18:10:37.552910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:37.552867 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m"] Oct 30 18:10:37.556641 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:10:37.556616 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod552542b1_fdd2_417f_88a5_614f087e6228.slice/crio-ab46474ac69b0857f6e7a111388ac3f37c5b20aa2c56cc463422efde06039759 WatchSource:0}: Error finding container ab46474ac69b0857f6e7a111388ac3f37c5b20aa2c56cc463422efde06039759: Status 404 returned error can't find the container with id ab46474ac69b0857f6e7a111388ac3f37c5b20aa2c56cc463422efde06039759 Oct 30 18:10:37.984466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:37.984431 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" event={"ID":"552542b1-fdd2-417f-88a5-614f087e6228","Type":"ContainerStarted","Data":"bb8129c4fc7a57e50f8b02a91fe453f809e049c8679ed3accda424d6388d2d89"} Oct 30 18:10:37.984927 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:37.984480 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" event={"ID":"552542b1-fdd2-417f-88a5-614f087e6228","Type":"ContainerStarted","Data":"ab46474ac69b0857f6e7a111388ac3f37c5b20aa2c56cc463422efde06039759"} Oct 30 18:10:37.984927 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:37.984571 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" Oct 30 18:10:37.986027 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:37.986005 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" event={"ID":"771db49d-214d-4901-8885-50e40d3d4d88","Type":"ContainerStarted","Data":"65835ba5475943c5565b97503b07cce5f2c3d989d2660d206d14340a1b7acf4a"} Oct 30 18:10:37.986166 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:37.986039 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" event={"ID":"771db49d-214d-4901-8885-50e40d3d4d88","Type":"ContainerStarted","Data":"52d9932d4a14eeb2f7fdf1018c180f3191ad25581c674a984a0a484b578f6af6"} Oct 30 18:10:37.986166 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:37.986065 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" Oct 30 18:10:38.073647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:38.073591 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" podStartSLOduration=2.073573617 podStartE2EDuration="2.073573617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:38.073388498 +0000 UTC m=+537.389808818" watchObservedRunningTime="2026-10-30 18:10:38.073573617 +0000 UTC m=+537.389993939" Oct 30 18:10:38.074055 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:38.074016 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" podStartSLOduration=2.074008575 podStartE2EDuration="2.074008575s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:38.030854829 +0000 UTC m=+537.347275186" watchObservedRunningTime="2026-10-30 18:10:38.074008575 +0000 UTC m=+537.390428895" Oct 30 18:10:38.936822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:38.936789 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw"] Oct 30 18:10:38.944000 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:38.943966 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" Oct 30 18:10:38.944713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:38.944687 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw"] Oct 30 18:10:38.996558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:38.993444 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-4ef6ac-dockercfg-rhmzx\"" Oct 30 18:10:39.055357 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:39.055324 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-4ef6ac\"" Oct 30 18:10:39.100949 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:39.100916 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9p2j\" (UniqueName: \"kubernetes.io/projected/7b743aee-2a49-44f2-aaae-24cd2e760827-kube-api-access-m9p2j\") pod \"payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw\" (UID: \"7b743aee-2a49-44f2-aaae-24cd2e760827\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" Oct 30 18:10:39.101370 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:39.101345 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b743aee-2a49-44f2-aaae-24cd2e760827-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw\" (UID: \"7b743aee-2a49-44f2-aaae-24cd2e760827\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" Oct 30 18:10:39.202564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:39.202458 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b743aee-2a49-44f2-aaae-24cd2e760827-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw\" (UID: \"7b743aee-2a49-44f2-aaae-24cd2e760827\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" Oct 30 18:10:39.202564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:39.202546 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9p2j\" (UniqueName: \"kubernetes.io/projected/7b743aee-2a49-44f2-aaae-24cd2e760827-kube-api-access-m9p2j\") pod \"payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw\" (UID: \"7b743aee-2a49-44f2-aaae-24cd2e760827\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" Oct 30 18:10:39.203130 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:39.203109 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b743aee-2a49-44f2-aaae-24cd2e760827-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw\" (UID: \"7b743aee-2a49-44f2-aaae-24cd2e760827\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" Oct 30 18:10:39.236265 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:39.236235 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9p2j\" (UniqueName: \"kubernetes.io/projected/7b743aee-2a49-44f2-aaae-24cd2e760827-kube-api-access-m9p2j\") pod \"payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw\" (UID: \"7b743aee-2a49-44f2-aaae-24cd2e760827\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" Oct 30 18:10:39.345854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:39.345815 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" Oct 30 18:10:39.641695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:39.641663 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw"] Oct 30 18:10:39.645278 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:10:39.645249 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b743aee_2a49_44f2_aaae_24cd2e760827.slice/crio-adbf7bb0994cec247cacf7e0cea5a3f3de2270a12327541b1bc20086d40226c6 WatchSource:0}: Error finding container adbf7bb0994cec247cacf7e0cea5a3f3de2270a12327541b1bc20086d40226c6: Status 404 returned error can't find the container with id adbf7bb0994cec247cacf7e0cea5a3f3de2270a12327541b1bc20086d40226c6 Oct 30 18:10:40.006799 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:40.006764 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" event={"ID":"7b743aee-2a49-44f2-aaae-24cd2e760827","Type":"ContainerStarted","Data":"db648949b8f07000f2740bbaef0f28e67df628c8f38d478ed8f54e55698dfa19"} Oct 30 18:10:40.007237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:40.006821 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" event={"ID":"7b743aee-2a49-44f2-aaae-24cd2e760827","Type":"ContainerStarted","Data":"adbf7bb0994cec247cacf7e0cea5a3f3de2270a12327541b1bc20086d40226c6"} Oct 30 18:10:40.007237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:40.006844 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" Oct 30 18:10:40.057694 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:40.057644 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" podStartSLOduration=2.057627802 podStartE2EDuration="2.057627802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:40.055772181 +0000 UTC m=+539.372192504" watchObservedRunningTime="2026-10-30 18:10:40.057627802 +0000 UTC m=+539.374048122" Oct 30 18:10:40.936480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:40.936445 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" Oct 30 18:10:41.935423 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:41.935389 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" Oct 30 18:10:42.937351 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:42.937319 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" Oct 30 18:10:42.937768 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:42.937393 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" Oct 30 18:10:43.964750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:43.964719 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" Oct 30 18:10:43.966396 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:43.966371 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" Oct 30 18:10:45.975961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:45.975934 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" Oct 30 18:10:47.068762 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:47.068734 5232 generic.go:356] "Generic (PLEG): container finished" podID="3480923d-8625-430e-819c-628e203c4946" containerID="2b789eacda30691b99761c368abbc2e45d23be596696dfda30a2a6e4ce8b8202" exitCode=0 Oct 30 18:10:47.069175 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:47.068814 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" event={"ID":"3480923d-8625-430e-819c-628e203c4946","Type":"ContainerDied","Data":"2b789eacda30691b99761c368abbc2e45d23be596696dfda30a2a6e4ce8b8202"} Oct 30 18:10:47.069215 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:47.069202 5232 scope.go:122] "RemoveContainer" containerID="2b789eacda30691b99761c368abbc2e45d23be596696dfda30a2a6e4ce8b8202" Oct 30 18:10:47.070595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:47.070568 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:10:47.070657 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:47.070633 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:10:47.070694 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:47.070668 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:10:48.076404 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.076369 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" event={"ID":"3480923d-8625-430e-819c-628e203c4946","Type":"ContainerStarted","Data":"bb8807f96e9f65c7314888bfdbd7e39e1caa4e17e44c27525b693df398c6d627"} Oct 30 18:10:48.381321 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.381240 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:48.381466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.381403 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 18:10:48.381466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.381448 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" podUID="3480923d-8625-430e-819c-628e203c4946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 18:10:48.599997 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.599961 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6"] Oct 30 18:10:48.602802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.602780 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.639814 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.634643 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"kube-root-ca.crt\"" Oct 30 18:10:48.639814 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.634737 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"openshift-service-ca.crt\"" Oct 30 18:10:48.639814 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.639050 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6"] Oct 30 18:10:48.643334 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.643300 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-model-cache\") pod \"simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.643446 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.643341 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-tmp-dir\") pod \"simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.643446 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.643372 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-home\") pod \"simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.643560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.643462 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzvvt\" (UniqueName: \"kubernetes.io/projected/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-kube-api-access-hzvvt\") pod \"simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.643606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.643560 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-dshm\") pod \"simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.643688 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.643661 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-tls-certs\") pod \"simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.646402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.646382 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"default-dockercfg-d99qs\"" Oct 30 18:10:48.646477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.646385 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"simulator-w2-fde2c5-kserve-self-signed-certs\"" Oct 30 18:10:48.744266 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.744230 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-model-cache\") pod \"simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.744266 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.744271 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-tmp-dir\") pod \"simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.744497 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.744296 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-home\") pod \"simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.744497 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.744371 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzvvt\" (UniqueName: \"kubernetes.io/projected/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-kube-api-access-hzvvt\") pod \"simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.744497 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.744441 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-dshm\") pod \"simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.744497 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.744473 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-tls-certs\") pod \"simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.744804 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.744773 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-model-cache\") pod \"simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.744890 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.744867 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-home\") pod \"simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.744994 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.744975 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-tmp-dir\") pod \"simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.746919 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.746888 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-dshm\") pod \"simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.747133 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.747115 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-tls-certs\") pod \"simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.763875 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.763852 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzvvt\" (UniqueName: \"kubernetes.io/projected/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-kube-api-access-hzvvt\") pod \"simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:48.942735 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:48.942644 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:49.004387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:49.004357 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" Oct 30 18:10:49.005105 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:49.005079 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" Oct 30 18:10:49.241899 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:49.241870 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6"] Oct 30 18:10:49.246641 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:10:49.246608 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb6a6cc05_a0c6_4ddf_bfc0_782f1f66620b.slice/crio-e401daa9ba21039600043953c9fbd9135354325c7a45b635ca16159599f630a5 WatchSource:0}: Error finding container e401daa9ba21039600043953c9fbd9135354325c7a45b635ca16159599f630a5: Status 404 returned error can't find the container with id e401daa9ba21039600043953c9fbd9135354325c7a45b635ca16159599f630a5 Oct 30 18:10:49.381269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:49.381233 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 18:10:49.381426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:49.381296 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" podUID="3480923d-8625-430e-819c-628e203c4946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 18:10:50.090038 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:50.089998 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" event={"ID":"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b","Type":"ContainerStarted","Data":"0baea61d2cdf1633f7fcc5db6c53bd3b70cf82be5427f4ceab0063e53d26b468"} Oct 30 18:10:50.090225 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:50.090055 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" event={"ID":"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b","Type":"ContainerStarted","Data":"e401daa9ba21039600043953c9fbd9135354325c7a45b635ca16159599f630a5"} Oct 30 18:10:50.144670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:50.144606 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podStartSLOduration=2.144588539 podStartE2EDuration="2.144588539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:10:50.142020259 +0000 UTC m=+549.458440580" watchObservedRunningTime="2026-10-30 18:10:50.144588539 +0000 UTC m=+549.461008861" Oct 30 18:10:50.380784 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:50.380700 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 18:10:50.380784 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:50.380751 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" podUID="3480923d-8625-430e-819c-628e203c4946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 18:10:50.943100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:50.943065 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:51.042548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:51.042350 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" Oct 30 18:10:51.380912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:51.380870 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 18:10:51.381305 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:51.380924 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" podUID="3480923d-8625-430e-819c-628e203c4946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 18:10:52.381203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:52.381167 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" start-of-body= Oct 30 18:10:52.381837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:52.381800 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" podUID="3480923d-8625-430e-819c-628e203c4946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": dial tcp 10.133.0.30:15021: connect: connection refused" Oct 30 18:10:53.496274 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:53.496193 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:53.496828 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:53.496737 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:53.497599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:53.497581 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:10:57.077037 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:10:57.076995 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:10:58.675310 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.654094 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w"] Oct 30 18:10:58.683133 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.683104 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.701398 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.701362 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w"] Oct 30 18:10:58.710474 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.710420 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"kube-root-ca.crt\"" Oct 30 18:10:58.722854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.722820 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"openshift-service-ca.crt\"" Oct 30 18:10:58.722854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.722843 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"default-dockercfg-zff9h\"" Oct 30 18:10:58.723076 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.722949 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"simulator-w1-fb2fa0-kserve-self-signed-certs\"" Oct 30 18:10:58.801997 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.801494 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-home\") pod \"simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.801997 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.801558 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-tmp-dir\") pod \"simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.801997 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.801589 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a41b45-1068-426b-bbf8-422e1b4980ba-tls-certs\") pod \"simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.801997 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.801744 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fltj\" (UniqueName: \"kubernetes.io/projected/05a41b45-1068-426b-bbf8-422e1b4980ba-kube-api-access-7fltj\") pod \"simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.801997 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.801787 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-dshm\") pod \"simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.801997 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.801840 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-model-cache\") pod \"simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.902636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.902588 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fltj\" (UniqueName: \"kubernetes.io/projected/05a41b45-1068-426b-bbf8-422e1b4980ba-kube-api-access-7fltj\") pod \"simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.902830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.902661 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-dshm\") pod \"simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.902830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.902717 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-model-cache\") pod \"simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.902830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.902750 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-home\") pod \"simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.902830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.902773 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-tmp-dir\") pod \"simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.902830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.902800 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a41b45-1068-426b-bbf8-422e1b4980ba-tls-certs\") pod \"simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.903925 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.903690 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-home\") pod \"simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.903925 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.903857 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-model-cache\") pod \"simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.904106 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.903932 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-tmp-dir\") pod \"simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.932029 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.931916 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-dshm\") pod \"simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.932190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.932159 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a41b45-1068-426b-bbf8-422e1b4980ba-tls-certs\") pod \"simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.932255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.932206 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fltj\" (UniqueName: \"kubernetes.io/projected/05a41b45-1068-426b-bbf8-422e1b4980ba-kube-api-access-7fltj\") pod \"simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:58.945174 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.943574 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:58.978818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:58.978787 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:59.026620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:59.026356 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:10:59.198033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:59.197945 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:10:59.366425 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:10:59.366387 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w"] Oct 30 18:10:59.372680 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:10:59.372645 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod05a41b45_1068_426b_bbf8_422e1b4980ba.slice/crio-9cf5a7a4fab40952a16071b1a99798c276d9aaaf1ccc8c43b750fda9dc6661fe WatchSource:0}: Error finding container 9cf5a7a4fab40952a16071b1a99798c276d9aaaf1ccc8c43b750fda9dc6661fe: Status 404 returned error can't find the container with id 9cf5a7a4fab40952a16071b1a99798c276d9aaaf1ccc8c43b750fda9dc6661fe Oct 30 18:11:00.193809 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:00.193761 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" event={"ID":"05a41b45-1068-426b-bbf8-422e1b4980ba","Type":"ContainerStarted","Data":"adb635b41935ea40e15bca8b12d68c6dd85e96f4fdb06f42b11a84755d464648"} Oct 30 18:11:00.194369 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:00.193821 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" event={"ID":"05a41b45-1068-426b-bbf8-422e1b4980ba","Type":"ContainerStarted","Data":"9cf5a7a4fab40952a16071b1a99798c276d9aaaf1ccc8c43b750fda9dc6661fe"} Oct 30 18:11:00.240116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:00.240055 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podStartSLOduration=2.240036606 podStartE2EDuration="2.240036606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:10:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:00.237430114 +0000 UTC m=+559.553850432" watchObservedRunningTime="2026-10-30 18:11:00.240036606 +0000 UTC m=+559.556456928" Oct 30 18:11:01.026477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:01.026445 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:11:02.206384 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:02.206355 5232 generic.go:356] "Generic (PLEG): container finished" podID="ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" containerID="18d030245395eb9dfb629d43f057cbbfd732280822d9d92f1debf184fe1e8296" exitCode=0 Oct 30 18:11:02.206851 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:02.206422 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" event={"ID":"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e","Type":"ContainerDied","Data":"18d030245395eb9dfb629d43f057cbbfd732280822d9d92f1debf184fe1e8296"} Oct 30 18:11:02.206851 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:02.206847 5232 scope.go:122] "RemoveContainer" containerID="18d030245395eb9dfb629d43f057cbbfd732280822d9d92f1debf184fe1e8296" Oct 30 18:11:02.208200 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:02.208181 5232 generic.go:356] "Generic (PLEG): container finished" podID="f8c0e348-2b8d-48e2-9ec0-a21a83db391e" containerID="c644ecb172cd9640e48f6b6b1a0f8c1dc380318e906d876cce36c4fecfb5055b" exitCode=0 Oct 30 18:11:02.208302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:02.208218 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:11:02.208302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:02.208256 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" event={"ID":"f8c0e348-2b8d-48e2-9ec0-a21a83db391e","Type":"ContainerDied","Data":"c644ecb172cd9640e48f6b6b1a0f8c1dc380318e906d876cce36c4fecfb5055b"} Oct 30 18:11:02.208302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:02.208297 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:11:02.208420 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:02.208333 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:11:02.208728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:02.208711 5232 scope.go:122] "RemoveContainer" containerID="c644ecb172cd9640e48f6b6b1a0f8c1dc380318e906d876cce36c4fecfb5055b" Oct 30 18:11:02.209851 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:02.209827 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:11:02.209955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:02.209883 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:11:02.209955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:02.209916 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:11:03.249534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:03.246769 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" event={"ID":"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e","Type":"ContainerStarted","Data":"3a47235c7e61b1fe8c5b3590996f2ce2e23a04972f2c8cd2d9821ae7f94b92f5"} Oct 30 18:11:03.264394 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:03.263716 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" event={"ID":"f8c0e348-2b8d-48e2-9ec0-a21a83db391e","Type":"ContainerStarted","Data":"f15ae61695576b22e3376adff368e02d9edb1454d65fd6cc186ba20f5147687d"} Oct 30 18:11:03.502060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:03.501969 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:11:03.502288 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:03.502120 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 18:11:03.502409 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:03.502328 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" podUID="f8c0e348-2b8d-48e2-9ec0-a21a83db391e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 18:11:03.898853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:03.898770 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:11:03.899003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:03.898948 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 30 18:11:03.899003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:03.898991 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" podUID="ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 30 18:11:04.501676 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:04.501626 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 18:11:04.502076 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:04.501702 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" podUID="f8c0e348-2b8d-48e2-9ec0-a21a83db391e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 18:11:04.899566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:04.899448 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 30 18:11:04.899566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:04.899544 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" podUID="ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 30 18:11:05.501708 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:05.501669 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 18:11:05.502224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:05.501725 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" podUID="f8c0e348-2b8d-48e2-9ec0-a21a83db391e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 18:11:05.899790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:05.899713 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 30 18:11:05.899985 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:05.899964 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" podUID="ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 30 18:11:06.501920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:06.501879 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 18:11:06.502309 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:06.501947 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" podUID="f8c0e348-2b8d-48e2-9ec0-a21a83db391e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 18:11:06.898971 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:06.898888 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 30 18:11:06.898971 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:06.898941 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" podUID="ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 30 18:11:07.501726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:07.501687 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" start-of-body= Oct 30 18:11:07.501886 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:07.501776 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" podUID="f8c0e348-2b8d-48e2-9ec0-a21a83db391e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": dial tcp 10.133.0.31:15021: connect: connection refused" Oct 30 18:11:07.899173 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:07.899090 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" start-of-body= Oct 30 18:11:07.899173 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:07.899146 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" podUID="ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": dial tcp 10.133.0.32:15021: connect: connection refused" Oct 30 18:11:08.549033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.549001 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:11:08.549398 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.549359 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:11:08.550651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.550631 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:11:08.910552 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.910436 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w"] Oct 30 18:11:08.918073 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.918044 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:11:08.918883 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.918496 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:08.918883 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.918551 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:11:08.919816 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.919788 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:11:08.930995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.930971 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w"] Oct 30 18:11:08.942654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.942636 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-4ef6ac-openshift-default-dockercfg-vxmbz\"" Oct 30 18:11:08.998749 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.998718 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:08.998903 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.998777 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:08.998903 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.998881 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:08.998987 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.998967 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:08.999023 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.998996 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-flqgw\" (UniqueName: \"kubernetes.io/projected/5d5ba31c-c8f3-4467-be05-fd146702503e-kube-api-access-flqgw\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:08.999067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.999056 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5ba31c-c8f3-4467-be05-fd146702503e-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:08.999162 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.999146 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:08.999230 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.999215 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:08.999362 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:08.999342 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.026952 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.026924 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:11:09.039469 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.039448 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:11:09.100370 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.100338 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.100370 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.100375 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.100644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.100416 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.100644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.100437 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-flqgw\" (UniqueName: \"kubernetes.io/projected/5d5ba31c-c8f3-4467-be05-fd146702503e-kube-api-access-flqgw\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.100644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.100534 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5ba31c-c8f3-4467-be05-fd146702503e-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.100644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.100574 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.100644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.100635 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.100916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.100671 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.100916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.100731 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.100916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.100855 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.101073 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.101040 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.101073 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.101056 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.101378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.101313 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.101500 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.101477 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5ba31c-c8f3-4467-be05-fd146702503e-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.102859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.102830 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.103148 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.103127 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.120528 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.120481 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.120753 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.120735 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-flqgw\" (UniqueName: \"kubernetes.io/projected/5d5ba31c-c8f3-4467-be05-fd146702503e-kube-api-access-flqgw\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.232115 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.232037 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:09.370704 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.370677 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:11:09.570614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.570578 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w"] Oct 30 18:11:09.581141 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.581105 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:11:09.581245 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.581177 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:11:09.581245 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:09.581204 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:11:10.366246 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:10.366213 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" event={"ID":"5d5ba31c-c8f3-4467-be05-fd146702503e","Type":"ContainerStarted","Data":"f2171dbcbfc5785dd2caf0edf1669bde96f145454078c94b61223c8e2d73fc95"} Oct 30 18:11:10.366745 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:10.366262 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" event={"ID":"5d5ba31c-c8f3-4467-be05-fd146702503e","Type":"ContainerStarted","Data":"d299c1cc392289a4c387ad436ba7a25e9dfd737ecd80419ff6cd44ccfeeff3df"} Oct 30 18:11:11.234558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:11.233802 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:11.236319 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:11.236296 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:11:11.236448 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:11.236349 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" podUID="5d5ba31c-c8f3-4467-be05-fd146702503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 18:11:12.233535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:12.232824 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:11:12.233535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:12.232886 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" podUID="5d5ba31c-c8f3-4467-be05-fd146702503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 18:11:13.234980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:13.234927 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:11:13.235496 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:13.235014 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" podUID="5d5ba31c-c8f3-4467-be05-fd146702503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 18:11:14.234972 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:14.234933 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:11:14.235123 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:14.235015 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" podUID="5d5ba31c-c8f3-4467-be05-fd146702503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 18:11:15.236537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:15.236038 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" start-of-body= Oct 30 18:11:15.236537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:15.236098 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" podUID="5d5ba31c-c8f3-4467-be05-fd146702503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": dial tcp 10.133.0.45:15021: connect: connection refused" Oct 30 18:11:16.239797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.239770 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:16.240292 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.240171 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:16.241131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.241108 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:11:16.241131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.241108 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" podStartSLOduration=8.241095325 podStartE2EDuration="8.241095325s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:10.43330122 +0000 UTC m=+569.749721566" watchObservedRunningTime="2026-10-30 18:11:16.241095325 +0000 UTC m=+575.557515657" Oct 30 18:11:16.241785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.241764 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t"] Oct 30 18:11:16.245064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.245044 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.267795 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.267346 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"openshift-service-ca.crt\"" Oct 30 18:11:16.267795 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.267346 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"simulator-w0-6ce5eb-kserve-self-signed-certs\"" Oct 30 18:11:16.267795 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.267377 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"default-dockercfg-j6p62\"" Oct 30 18:11:16.267795 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.267404 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-6ce5eb\"/\"kube-root-ca.crt\"" Oct 30 18:11:16.270166 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.270143 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t"] Oct 30 18:11:16.299957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.299934 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-dshm\") pod \"simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.300076 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.300045 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-tmp-dir\") pod \"simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.300076 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.300068 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7634ee23-ee09-4bd9-92a1-4a56b4a82460-tls-certs\") pod \"simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.300197 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.300176 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-model-cache\") pod \"simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.300254 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.300215 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bfpnt\" (UniqueName: \"kubernetes.io/projected/7634ee23-ee09-4bd9-92a1-4a56b4a82460-kube-api-access-bfpnt\") pod \"simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.300587 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.300464 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-home\") pod \"simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.401643 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.401602 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-model-cache\") pod \"simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.401826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.401653 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bfpnt\" (UniqueName: \"kubernetes.io/projected/7634ee23-ee09-4bd9-92a1-4a56b4a82460-kube-api-access-bfpnt\") pod \"simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.401826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.401719 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-home\") pod \"simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.401955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.401824 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-dshm\") pod \"simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.401955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.401894 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-tmp-dir\") pod \"simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.401955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.401918 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7634ee23-ee09-4bd9-92a1-4a56b4a82460-tls-certs\") pod \"simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.402105 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.402052 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-model-cache\") pod \"simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.402379 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.402355 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-tmp-dir\") pod \"simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.402591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.402454 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-home\") pod \"simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.405537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.405479 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-dshm\") pod \"simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.407659 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.407631 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7634ee23-ee09-4bd9-92a1-4a56b4a82460-tls-certs\") pod \"simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.420990 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.420968 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bfpnt\" (UniqueName: \"kubernetes.io/projected/7634ee23-ee09-4bd9-92a1-4a56b4a82460-kube-api-access-bfpnt\") pod \"simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.558856 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.558774 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:16.855537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:16.855434 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t"] Oct 30 18:11:16.859588 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:11:16.859552 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod7634ee23_ee09_4bd9_92a1_4a56b4a82460.slice/crio-907c8b95e592c711b91b7ea82dc32cb571c657b0c2508a65617703e41b33d365 WatchSource:0}: Error finding container 907c8b95e592c711b91b7ea82dc32cb571c657b0c2508a65617703e41b33d365: Status 404 returned error can't find the container with id 907c8b95e592c711b91b7ea82dc32cb571c657b0c2508a65617703e41b33d365 Oct 30 18:11:17.404093 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:17.404052 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" event={"ID":"7634ee23-ee09-4bd9-92a1-4a56b4a82460","Type":"ContainerStarted","Data":"2ad1f5acf196c28ab1eca94aff725423e5403c5269b67ec2d8c9b45a683c6b13"} Oct 30 18:11:17.404570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:17.404105 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" event={"ID":"7634ee23-ee09-4bd9-92a1-4a56b4a82460","Type":"ContainerStarted","Data":"907c8b95e592c711b91b7ea82dc32cb571c657b0c2508a65617703e41b33d365"} Oct 30 18:11:17.446323 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:17.446278 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podStartSLOduration=1.446264047 podStartE2EDuration="1.446264047s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:17.444888568 +0000 UTC m=+576.761308890" watchObservedRunningTime="2026-10-30 18:11:17.446264047 +0000 UTC m=+576.762684367" Oct 30 18:11:17.566339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:17.563269 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:26.559850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:26.559817 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:26.572418 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:26.572396 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:27.035762 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:11:27.035727 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:11:27.479127 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:27.479051 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:11:29.655969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.655937 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2"] Oct 30 18:11:29.659693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.659676 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.688938 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.688908 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"openshift-service-ca.crt\"" Oct 30 18:11:29.688938 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.688908 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"kube-root-ca.crt\"" Oct 30 18:11:29.689168 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.688976 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"default-dockercfg-vlhvr\"" Oct 30 18:11:29.689168 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.689065 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"simulator-w4-4ef6ac-kserve-self-signed-certs\"" Oct 30 18:11:29.691390 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.691364 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2"] Oct 30 18:11:29.755108 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.755076 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-model-cache\") pod \"simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.755108 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.755113 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-tmp-dir\") pod \"simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.755306 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.755143 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6x58\" (UniqueName: \"kubernetes.io/projected/52075caf-2e4a-4c47-9a7a-9a9fcc744277-kube-api-access-x6x58\") pod \"simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.755306 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.755251 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52075caf-2e4a-4c47-9a7a-9a9fcc744277-tls-certs\") pod \"simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.755376 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.755305 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-dshm\") pod \"simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.755376 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.755369 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-home\") pod \"simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.856494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.856458 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6x58\" (UniqueName: \"kubernetes.io/projected/52075caf-2e4a-4c47-9a7a-9a9fcc744277-kube-api-access-x6x58\") pod \"simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.856698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.856561 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52075caf-2e4a-4c47-9a7a-9a9fcc744277-tls-certs\") pod \"simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.856698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.856588 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-dshm\") pod \"simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.856698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.856627 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-home\") pod \"simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.856698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.856693 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-model-cache\") pod \"simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.856912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.856728 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-tmp-dir\") pod \"simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.857080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.857058 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-model-cache\") pod \"simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.857155 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.857096 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-home\") pod \"simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.857155 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.857133 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-tmp-dir\") pod \"simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.858977 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.858961 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-dshm\") pod \"simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.859263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.859247 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52075caf-2e4a-4c47-9a7a-9a9fcc744277-tls-certs\") pod \"simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.875481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.875455 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6x58\" (UniqueName: \"kubernetes.io/projected/52075caf-2e4a-4c47-9a7a-9a9fcc744277-kube-api-access-x6x58\") pod \"simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:29.973086 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:29.972996 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:30.269089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:30.269062 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2"] Oct 30 18:11:30.272309 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:11:30.272280 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod52075caf_2e4a_4c47_9a7a_9a9fcc744277.slice/crio-3409526554e35b29a5afc0e90221008a2614cdcea1a8d16325be3d8a08246af0 WatchSource:0}: Error finding container 3409526554e35b29a5afc0e90221008a2614cdcea1a8d16325be3d8a08246af0: Status 404 returned error can't find the container with id 3409526554e35b29a5afc0e90221008a2614cdcea1a8d16325be3d8a08246af0 Oct 30 18:11:30.485444 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:30.485414 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" event={"ID":"52075caf-2e4a-4c47-9a7a-9a9fcc744277","Type":"ContainerStarted","Data":"6b6b52f44ba1e7d8d967263c7bb390e46b52144b0b070185539fd21f7db63271"} Oct 30 18:11:30.485647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:30.485458 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" event={"ID":"52075caf-2e4a-4c47-9a7a-9a9fcc744277","Type":"ContainerStarted","Data":"3409526554e35b29a5afc0e90221008a2614cdcea1a8d16325be3d8a08246af0"} Oct 30 18:11:30.524645 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:30.524558 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podStartSLOduration=1.524543083 podStartE2EDuration="1.524543083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:30.524125779 +0000 UTC m=+589.840546100" watchObservedRunningTime="2026-10-30 18:11:30.524543083 +0000 UTC m=+589.840963396" Oct 30 18:11:30.973172 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:30.973104 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:39.973623 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:39.973592 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:39.986306 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:39.986280 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:40.553788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:40.553761 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:11:52.985012 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:52.984978 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k"] Oct 30 18:11:52.989335 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:52.989304 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.006813 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.006786 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k"] Oct 30 18:11:53.016981 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.016941 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8b6cf1\"/\"kube-root-ca.crt\"" Oct 30 18:11:53.016981 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.016963 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-8b6cf1\"/\"openshift-service-ca.crt\"" Oct 30 18:11:53.019070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.019051 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8b6cf1\"/\"premium-w3-8b6cf1-kserve-self-signed-certs\"" Oct 30 18:11:53.019141 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.019056 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8b6cf1\"/\"default-dockercfg-jl62g\"" Oct 30 18:11:53.115967 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.115936 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-model-cache\") pod \"premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.115967 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.115968 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tjr57\" (UniqueName: \"kubernetes.io/projected/1a0e1051-f784-4141-a8fa-c5fb432f4abd-kube-api-access-tjr57\") pod \"premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.116160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.115998 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-home\") pod \"premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.116160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.116062 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-dshm\") pod \"premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.116160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.116119 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-tmp-dir\") pod \"premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.116272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.116187 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a0e1051-f784-4141-a8fa-c5fb432f4abd-tls-certs\") pod \"premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.216847 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.216815 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-model-cache\") pod \"premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.216847 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.216846 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tjr57\" (UniqueName: \"kubernetes.io/projected/1a0e1051-f784-4141-a8fa-c5fb432f4abd-kube-api-access-tjr57\") pod \"premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.217055 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.216875 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-home\") pod \"premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.217055 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.216992 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-dshm\") pod \"premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.217168 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.217070 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-tmp-dir\") pod \"premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.217168 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.217118 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a0e1051-f784-4141-a8fa-c5fb432f4abd-tls-certs\") pod \"premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.217268 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.217179 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-model-cache\") pod \"premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.217268 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.217234 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-home\") pod \"premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.217402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.217385 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-tmp-dir\") pod \"premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.219345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.219319 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-dshm\") pod \"premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.219754 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.219728 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a0e1051-f784-4141-a8fa-c5fb432f4abd-tls-certs\") pod \"premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.235390 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.235334 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd"] Oct 30 18:11:53.239353 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.239336 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.240457 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.240432 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tjr57\" (UniqueName: \"kubernetes.io/projected/1a0e1051-f784-4141-a8fa-c5fb432f4abd-kube-api-access-tjr57\") pod \"premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.257616 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.257593 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-8b6cf1-openshift-default-dockercfg-l4bh4\"" Oct 30 18:11:53.267248 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.267215 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd"] Oct 30 18:11:53.302292 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.302263 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:53.318155 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.318129 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.318275 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.318164 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.318275 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.318184 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.318275 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.318264 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.318444 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.318291 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.318444 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.318378 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.318444 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.318407 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.318651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.318484 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnqtx\" (UniqueName: \"kubernetes.io/projected/f9270d40-a0b1-4c84-a28b-9f4bca0487be-kube-api-access-rnqtx\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.318651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.318575 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.419545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.419492 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.419729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.419551 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.419729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.419569 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.419729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.419594 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.419729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.419682 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.419967 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.419756 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.419967 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.419807 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.419967 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.419844 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnqtx\" (UniqueName: \"kubernetes.io/projected/f9270d40-a0b1-4c84-a28b-9f4bca0487be-kube-api-access-rnqtx\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.419967 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.419872 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.420181 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.420005 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.420181 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.420033 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.420472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.420383 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.420472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.420458 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.420691 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.420593 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.422109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.422088 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.422378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.422362 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.439852 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.439828 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.440115 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.440094 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnqtx\" (UniqueName: \"kubernetes.io/projected/f9270d40-a0b1-4c84-a28b-9f4bca0487be-kube-api-access-rnqtx\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.567218 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.567186 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:53.601965 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.601636 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k"] Oct 30 18:11:53.604933 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:11:53.604887 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1a0e1051_f784_4141_a8fa_c5fb432f4abd.slice/crio-2a6e79ad5728b076950067441e2a80cbe10368e226d43e379313cb0c90687705 WatchSource:0}: Error finding container 2a6e79ad5728b076950067441e2a80cbe10368e226d43e379313cb0c90687705: Status 404 returned error can't find the container with id 2a6e79ad5728b076950067441e2a80cbe10368e226d43e379313cb0c90687705 Oct 30 18:11:53.628946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.627885 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" event={"ID":"1a0e1051-f784-4141-a8fa-c5fb432f4abd","Type":"ContainerStarted","Data":"2a6e79ad5728b076950067441e2a80cbe10368e226d43e379313cb0c90687705"} Oct 30 18:11:53.758346 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.758311 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm"] Oct 30 18:11:53.762915 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.762895 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.780443 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.780421 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-fdf4da-openshift-default-dockercfg-gnh4d\"" Oct 30 18:11:53.781628 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.781603 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm"] Oct 30 18:11:53.824368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.824293 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.824368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.824352 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.824580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.824438 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.824580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.824464 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shdvd\" (UniqueName: \"kubernetes.io/projected/a337e101-6a41-4416-9731-f28cf2afd0f2-kube-api-access-shdvd\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.824580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.824565 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a337e101-6a41-4416-9731-f28cf2afd0f2-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.824682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.824602 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.824682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.824619 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.824742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.824719 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.824742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.824736 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.878119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.878094 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd"] Oct 30 18:11:53.881022 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:11:53.880996 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf9270d40_a0b1_4c84_a28b_9f4bca0487be.slice/crio-f17c47f7938a36bf90998a1ecde43046f937a0111f8180305a1d1532901d57f8 WatchSource:0}: Error finding container f17c47f7938a36bf90998a1ecde43046f937a0111f8180305a1d1532901d57f8: Status 404 returned error can't find the container with id f17c47f7938a36bf90998a1ecde43046f937a0111f8180305a1d1532901d57f8 Oct 30 18:11:53.883533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.883462 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:11:53.883603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.883587 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:11:53.883648 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.883632 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:11:53.925541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.925497 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a337e101-6a41-4416-9731-f28cf2afd0f2-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.925665 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.925569 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.925665 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.925595 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.925665 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.925643 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.925824 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.925670 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.925824 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.925776 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.925927 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.925875 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.926009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.925974 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.926069 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.926032 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shdvd\" (UniqueName: \"kubernetes.io/projected/a337e101-6a41-4416-9731-f28cf2afd0f2-kube-api-access-shdvd\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.926565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.926255 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a337e101-6a41-4416-9731-f28cf2afd0f2-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.926565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.925982 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.926565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.926531 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.926827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.926805 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.926970 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.926940 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.928419 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.928401 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.928564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.928468 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.946175 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.946069 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:53.946229 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:53.946217 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shdvd\" (UniqueName: \"kubernetes.io/projected/a337e101-6a41-4416-9731-f28cf2afd0f2-kube-api-access-shdvd\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:54.022204 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.022167 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr"] Oct 30 18:11:54.025757 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.025736 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.048037 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.048004 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr"] Oct 30 18:11:54.076274 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.076197 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:54.129209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.128100 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.129209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.128210 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b66639f-9863-4681-b052-af4d1007c74b-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.129209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.128248 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b66639f-9863-4681-b052-af4d1007c74b-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.129209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.128325 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.129209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.128369 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b66639f-9863-4681-b052-af4d1007c74b-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.129209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.128428 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.129209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.128473 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.129209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.128541 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g5rx7\" (UniqueName: \"kubernetes.io/projected/5b66639f-9863-4681-b052-af4d1007c74b-kube-api-access-g5rx7\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.129209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.128573 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.229196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.229163 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.229196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.229201 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b66639f-9863-4681-b052-af4d1007c74b-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.229445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.229227 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.229445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.229264 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.229445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.229311 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g5rx7\" (UniqueName: \"kubernetes.io/projected/5b66639f-9863-4681-b052-af4d1007c74b-kube-api-access-g5rx7\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.229445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.229344 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.229445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.229390 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.229766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.229473 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b66639f-9863-4681-b052-af4d1007c74b-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.229766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.229549 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b66639f-9863-4681-b052-af4d1007c74b-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.229766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.229654 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.229766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.229662 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.229961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.229879 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.229961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.229912 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.230355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.230330 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b66639f-9863-4681-b052-af4d1007c74b-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.232200 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.232174 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.232643 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.232625 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b66639f-9863-4681-b052-af4d1007c74b-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.248621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.248590 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b66639f-9863-4681-b052-af4d1007c74b-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.248744 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.248671 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g5rx7\" (UniqueName: \"kubernetes.io/projected/5b66639f-9863-4681-b052-af4d1007c74b-kube-api-access-g5rx7\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.343035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.342722 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:54.419298 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.418714 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm"] Oct 30 18:11:54.432014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.431972 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:11:54.432247 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.432222 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:11:54.432368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.432347 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:11:54.667259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.667161 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" event={"ID":"f9270d40-a0b1-4c84-a28b-9f4bca0487be","Type":"ContainerStarted","Data":"2a4dc2cd673a83c04b437f0a2a42693769812dfff21b4feffd956f4ab3000743"} Oct 30 18:11:54.667419 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.667268 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" event={"ID":"f9270d40-a0b1-4c84-a28b-9f4bca0487be","Type":"ContainerStarted","Data":"f17c47f7938a36bf90998a1ecde43046f937a0111f8180305a1d1532901d57f8"} Oct 30 18:11:54.669196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.669169 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" event={"ID":"1a0e1051-f784-4141-a8fa-c5fb432f4abd","Type":"ContainerStarted","Data":"b81d1ff926caddabf97a8403c2ef6a8a4fc4681483a8948773e9cf2e63574061"} Oct 30 18:11:54.671020 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.670994 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" event={"ID":"a337e101-6a41-4416-9731-f28cf2afd0f2","Type":"ContainerStarted","Data":"c071be777aa99da8d1f7fc20a696da637a9d2e2b43b779f2e7d1dd3d10047766"} Oct 30 18:11:54.671111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.671049 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" event={"ID":"a337e101-6a41-4416-9731-f28cf2afd0f2","Type":"ContainerStarted","Data":"5f12c0230e3b55c321c10f4457e359da2923e4cb26431590b623716eb1afcc3c"} Oct 30 18:11:54.694530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.694479 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr"] Oct 30 18:11:54.699692 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:11:54.699659 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b66639f_9863_4681_b052_af4d1007c74b.slice/crio-e442b8c79bbbaf9af552b9bb0e36437376ba2f6ec17b7552823123c202950d5c WatchSource:0}: Error finding container e442b8c79bbbaf9af552b9bb0e36437376ba2f6ec17b7552823123c202950d5c: Status 404 returned error can't find the container with id e442b8c79bbbaf9af552b9bb0e36437376ba2f6ec17b7552823123c202950d5c Oct 30 18:11:54.702366 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.702331 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:11:54.702467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.702411 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:11:54.702467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.702458 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:11:54.757022 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.756957 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" podStartSLOduration=1.756934196 podStartE2EDuration="1.756934196s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:54.713070485 +0000 UTC m=+614.029490808" watchObservedRunningTime="2026-10-30 18:11:54.756934196 +0000 UTC m=+614.073354518" Oct 30 18:11:54.798818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.798765 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podStartSLOduration=2.79874912 podStartE2EDuration="2.79874912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:54.758202197 +0000 UTC m=+614.074622520" watchObservedRunningTime="2026-10-30 18:11:54.79874912 +0000 UTC m=+614.115169441" Oct 30 18:11:54.799402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:54.799354 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" podStartSLOduration=1.799343787 podStartE2EDuration="1.799343787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:54.798136428 +0000 UTC m=+614.114556750" watchObservedRunningTime="2026-10-30 18:11:54.799343787 +0000 UTC m=+614.115764109" Oct 30 18:11:55.077191 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:55.077153 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:11:55.078782 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:55.078752 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:11:55.078844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:55.078818 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" podUID="a337e101-6a41-4416-9731-f28cf2afd0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 18:11:55.302803 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:55.302766 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:11:55.567556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:55.567519 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:11:55.568992 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:55.568972 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 18:11:55.569063 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:55.569013 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" podUID="f9270d40-a0b1-4c84-a28b-9f4bca0487be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 18:11:55.681375 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:55.681340 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" event={"ID":"5b66639f-9863-4681-b052-af4d1007c74b","Type":"ContainerStarted","Data":"2ce34298f678e487bfdec256893fdd00fc11a25626758ada2f037b01d858abd0"} Oct 30 18:11:55.681566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:55.681394 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" event={"ID":"5b66639f-9863-4681-b052-af4d1007c74b","Type":"ContainerStarted","Data":"e442b8c79bbbaf9af552b9bb0e36437376ba2f6ec17b7552823123c202950d5c"} Oct 30 18:11:55.772562 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:55.772395 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" podStartSLOduration=2.772373115 podStartE2EDuration="2.772373115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:11:55.749141758 +0000 UTC m=+615.065562080" watchObservedRunningTime="2026-10-30 18:11:55.772373115 +0000 UTC m=+615.088793438" Oct 30 18:11:56.077945 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:56.077905 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:11:56.078606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:56.078559 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" podUID="a337e101-6a41-4416-9731-f28cf2afd0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 18:11:56.360088 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:56.353908 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:11:56.378547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:56.370166 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 18:11:56.378547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:56.370231 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" podUID="5b66639f-9863-4681-b052-af4d1007c74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 18:11:56.595556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:56.583646 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 18:11:56.595556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:56.583706 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" podUID="f9270d40-a0b1-4c84-a28b-9f4bca0487be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 18:11:57.089560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:57.078624 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:11:57.089560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:57.078691 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" podUID="a337e101-6a41-4416-9731-f28cf2afd0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 18:11:57.108054 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:11:57.108013 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:11:57.348035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:57.347942 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 18:11:57.349584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:57.349545 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" podUID="5b66639f-9863-4681-b052-af4d1007c74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 18:11:57.576640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:57.576605 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 18:11:57.576937 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:57.576903 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" podUID="f9270d40-a0b1-4c84-a28b-9f4bca0487be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 18:11:58.080009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:58.079974 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:11:58.080298 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:58.080274 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" podUID="a337e101-6a41-4416-9731-f28cf2afd0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 18:11:58.348598 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:58.344912 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 18:11:58.348598 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:58.344976 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" podUID="5b66639f-9863-4681-b052-af4d1007c74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 18:11:58.580535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:58.571603 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 18:11:58.580535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:58.571665 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" podUID="f9270d40-a0b1-4c84-a28b-9f4bca0487be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 18:11:59.077535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:59.076885 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:11:59.077535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:59.076951 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" podUID="a337e101-6a41-4416-9731-f28cf2afd0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 18:11:59.350619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:59.350198 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 18:11:59.350619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:59.350261 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" podUID="5b66639f-9863-4681-b052-af4d1007c74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 18:11:59.609370 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:59.609238 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 18:11:59.634598 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:11:59.609739 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" podUID="f9270d40-a0b1-4c84-a28b-9f4bca0487be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 18:12:00.078364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:00.077922 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:12:00.078364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:00.077985 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" podUID="a337e101-6a41-4416-9731-f28cf2afd0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 18:12:00.348588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:00.347922 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 18:12:00.348588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:00.347984 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" podUID="5b66639f-9863-4681-b052-af4d1007c74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 18:12:00.573419 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:00.573385 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" start-of-body= Oct 30 18:12:00.574060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:00.574014 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" podUID="f9270d40-a0b1-4c84-a28b-9f4bca0487be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": dial tcp 10.133.0.49:15021: connect: connection refused" Oct 30 18:12:01.085535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:01.078300 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" start-of-body= Oct 30 18:12:01.085535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:01.078364 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" podUID="a337e101-6a41-4416-9731-f28cf2afd0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": dial tcp 10.133.0.50:15021: connect: connection refused" Oct 30 18:12:01.344212 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:01.344120 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" start-of-body= Oct 30 18:12:01.344212 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:01.344189 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" podUID="5b66639f-9863-4681-b052-af4d1007c74b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.51:15021/healthz/ready\": dial tcp 10.133.0.51:15021: connect: connection refused" Oct 30 18:12:01.572312 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:01.572274 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:12:01.572834 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:01.572806 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:12:01.574063 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:01.574037 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:12:02.350954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:02.350919 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:12:02.351390 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:02.351363 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:12:02.352117 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:02.352096 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:12:02.388015 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:02.387990 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:12:02.388308 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:02.388288 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:12:02.390227 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:02.390203 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:12:03.303146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:03.303113 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:12:03.315671 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:03.315647 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:12:03.848671 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:03.848640 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:12:08.656989 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.656951 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst"] Oct 30 18:12:08.661138 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.661116 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.676937 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.676915 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-fb2fa0-openshift-default-dockercfg-vw5sp\"" Oct 30 18:12:08.692280 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.692251 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst"] Oct 30 18:12:08.794742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.794707 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.794919 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.794757 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.794919 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.794866 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.795068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.794999 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.795068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.795048 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rm9t\" (UniqueName: \"kubernetes.io/projected/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-kube-api-access-2rm9t\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.795258 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.795165 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.795258 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.795219 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.795339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.795283 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.795339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.795304 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.896676 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.896644 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.896837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.896807 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rm9t\" (UniqueName: \"kubernetes.io/projected/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-kube-api-access-2rm9t\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.896934 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.896918 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.897028 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.897009 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.897081 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.897068 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.897133 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.897093 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.897190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.897140 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.897190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.897173 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.897302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.897220 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.897468 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.897442 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.897644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.897618 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.897734 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.897665 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.897802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.897733 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.898028 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.898005 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.899572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.899544 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.899668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.899655 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.918080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.918022 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rm9t\" (UniqueName: \"kubernetes.io/projected/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-kube-api-access-2rm9t\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.918080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.918027 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:08.974391 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:08.974364 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:09.292087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:09.292056 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst"] Oct 30 18:12:09.295941 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:12:09.295918 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc88a6b3a_815b_4dfd_bb64_54f1c530e3fe.slice/crio-57d057e779e6c5022b9a368799d84241ef6cf3f0582c83b40b3a241e336bb857 WatchSource:0}: Error finding container 57d057e779e6c5022b9a368799d84241ef6cf3f0582c83b40b3a241e336bb857: Status 404 returned error can't find the container with id 57d057e779e6c5022b9a368799d84241ef6cf3f0582c83b40b3a241e336bb857 Oct 30 18:12:09.298330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:09.298297 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:12:09.298397 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:09.298363 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:12:09.298397 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:09.298392 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:12:09.872414 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:09.872377 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" event={"ID":"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe","Type":"ContainerStarted","Data":"e051c9b75058bc0f431667c0276eb501b4f275b14caceeb11e56e95b28a84310"} Oct 30 18:12:09.872838 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:09.872429 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" event={"ID":"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe","Type":"ContainerStarted","Data":"57d057e779e6c5022b9a368799d84241ef6cf3f0582c83b40b3a241e336bb857"} Oct 30 18:12:09.974808 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:09.974775 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:10.977678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:10.977641 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:12:10.978262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:10.978228 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" podUID="c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 18:12:11.975309 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:11.975275 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:12:11.975476 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:11.975326 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" podUID="c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 18:12:12.975424 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:12.975389 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:12:12.975932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:12.975441 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" podUID="c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 18:12:13.975414 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:13.975385 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:12:13.975607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:13.975445 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" podUID="c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 18:12:14.975624 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:14.975584 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:12:14.976034 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:14.975656 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" podUID="c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 18:12:15.986536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:15.981939 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" start-of-body= Oct 30 18:12:15.986536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:15.982005 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" podUID="c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.52:15021/healthz/ready\": dial tcp 10.133.0.52:15021: connect: connection refused" Oct 30 18:12:17.411315 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:17.411286 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:17.411853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:17.411714 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:17.413080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:17.413058 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:12:17.466958 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:17.466904 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" podStartSLOduration=9.466886303 podStartE2EDuration="9.466886303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:09.938681669 +0000 UTC m=+629.255101990" watchObservedRunningTime="2026-10-30 18:12:17.466886303 +0000 UTC m=+636.783306623" Oct 30 18:12:27.033632 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:12:27.033549 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:12:35.104429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:35.104386 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb"] Oct 30 18:12:35.112002 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:35.111975 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb" Oct 30 18:12:35.124486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:35.124460 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb"] Oct 30 18:12:35.133062 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:35.133037 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:12:35.133195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:35.133079 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:12:35.176970 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:35.176934 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17f67c08-cc43-45c3-995f-ed8c7e4c7708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ab538b5-wbkkb\" (UID: \"17f67c08-cc43-45c3-995f-ed8c7e4c7708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb" Oct 30 18:12:35.176970 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:35.176977 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k9vk5\" (UniqueName: \"kubernetes.io/projected/17f67c08-cc43-45c3-995f-ed8c7e4c7708-kube-api-access-k9vk5\") pod \"maas-api-revoke-sub-7ab538b5-wbkkb\" (UID: \"17f67c08-cc43-45c3-995f-ed8c7e4c7708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb" Oct 30 18:12:35.177179 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:35.177083 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17f67c08-cc43-45c3-995f-ed8c7e4c7708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ab538b5-wbkkb\" (UID: \"17f67c08-cc43-45c3-995f-ed8c7e4c7708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb" Oct 30 18:12:35.278365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:35.278332 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17f67c08-cc43-45c3-995f-ed8c7e4c7708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ab538b5-wbkkb\" (UID: \"17f67c08-cc43-45c3-995f-ed8c7e4c7708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb" Oct 30 18:12:35.278563 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:35.278377 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k9vk5\" (UniqueName: \"kubernetes.io/projected/17f67c08-cc43-45c3-995f-ed8c7e4c7708-kube-api-access-k9vk5\") pod \"maas-api-revoke-sub-7ab538b5-wbkkb\" (UID: \"17f67c08-cc43-45c3-995f-ed8c7e4c7708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb" Oct 30 18:12:35.278563 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:35.278421 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17f67c08-cc43-45c3-995f-ed8c7e4c7708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ab538b5-wbkkb\" (UID: \"17f67c08-cc43-45c3-995f-ed8c7e4c7708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb" Oct 30 18:12:35.279082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:35.279058 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17f67c08-cc43-45c3-995f-ed8c7e4c7708-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7ab538b5-wbkkb\" (UID: \"17f67c08-cc43-45c3-995f-ed8c7e4c7708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb" Oct 30 18:12:35.280900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:35.280877 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17f67c08-cc43-45c3-995f-ed8c7e4c7708-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7ab538b5-wbkkb\" (UID: \"17f67c08-cc43-45c3-995f-ed8c7e4c7708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb" Oct 30 18:12:35.297413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:35.297384 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k9vk5\" (UniqueName: \"kubernetes.io/projected/17f67c08-cc43-45c3-995f-ed8c7e4c7708-kube-api-access-k9vk5\") pod \"maas-api-revoke-sub-7ab538b5-wbkkb\" (UID: \"17f67c08-cc43-45c3-995f-ed8c7e4c7708\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb" Oct 30 18:12:35.427971 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:35.427885 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb" Oct 30 18:12:35.725171 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:35.725088 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb"] Oct 30 18:12:35.728725 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:12:35.728695 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f67c08_cc43_45c3_995f_ed8c7e4c7708.slice/crio-b80321a22f84d51ea949bbea8e5293ba5d3d48b13a010720a66fbbc8ff414d00 WatchSource:0}: Error finding container b80321a22f84d51ea949bbea8e5293ba5d3d48b13a010720a66fbbc8ff414d00: Status 404 returned error can't find the container with id b80321a22f84d51ea949bbea8e5293ba5d3d48b13a010720a66fbbc8ff414d00 Oct 30 18:12:35.731171 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:35.731155 5232 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:12:36.046186 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:36.046154 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb" event={"ID":"17f67c08-cc43-45c3-995f-ed8c7e4c7708","Type":"ContainerStarted","Data":"b80321a22f84d51ea949bbea8e5293ba5d3d48b13a010720a66fbbc8ff414d00"} Oct 30 18:12:39.065451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:39.065419 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb" event={"ID":"17f67c08-cc43-45c3-995f-ed8c7e4c7708","Type":"ContainerStarted","Data":"7a181cb1ab898eeada4c2f7a2fc5d3024c9fcddffa8b4ae423965eff540b2734"} Oct 30 18:12:39.103607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:39.103550 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb" podStartSLOduration=1.558004537 podStartE2EDuration="4.103535481s" totalImagesPullingTime="2.545530944s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:39.101907436 +0000 UTC m=+658.418327758" watchObservedRunningTime="2026-10-30 18:12:39.103535481 +0000 UTC m=+658.419955813" Oct 30 18:12:40.072347 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:40.072307 5232 generic.go:356] "Generic (PLEG): container finished" podID="17f67c08-cc43-45c3-995f-ed8c7e4c7708" containerID="7a181cb1ab898eeada4c2f7a2fc5d3024c9fcddffa8b4ae423965eff540b2734" exitCode=0 Oct 30 18:12:40.072809 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:40.072397 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb" event={"ID":"17f67c08-cc43-45c3-995f-ed8c7e4c7708","Type":"ContainerDied","Data":"7a181cb1ab898eeada4c2f7a2fc5d3024c9fcddffa8b4ae423965eff540b2734"} Oct 30 18:12:41.226399 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:41.226372 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb" Oct 30 18:12:41.268241 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:41.268214 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f67c08-cc43-45c3-995f-ed8c7e4c7708-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17f67c08-cc43-45c3-995f-ed8c7e4c7708-maas-api-openshift-ca\") pod \"17f67c08-cc43-45c3-995f-ed8c7e4c7708\" (UID: \"17f67c08-cc43-45c3-995f-ed8c7e4c7708\") " Oct 30 18:12:41.268397 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:41.268245 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17f67c08-cc43-45c3-995f-ed8c7e4c7708-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17f67c08-cc43-45c3-995f-ed8c7e4c7708-maas-api-xks-ca\") pod \"17f67c08-cc43-45c3-995f-ed8c7e4c7708\" (UID: \"17f67c08-cc43-45c3-995f-ed8c7e4c7708\") " Oct 30 18:12:41.268397 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:41.268297 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f67c08-cc43-45c3-995f-ed8c7e4c7708-kube-api-access-k9vk5\" (UniqueName: \"kubernetes.io/projected/17f67c08-cc43-45c3-995f-ed8c7e4c7708-kube-api-access-k9vk5\") pod \"17f67c08-cc43-45c3-995f-ed8c7e4c7708\" (UID: \"17f67c08-cc43-45c3-995f-ed8c7e4c7708\") " Oct 30 18:12:41.268618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:41.268589 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f67c08-cc43-45c3-995f-ed8c7e4c7708-maas-api-openshift-ca" pod "17f67c08-cc43-45c3-995f-ed8c7e4c7708" (UID: "17f67c08-cc43-45c3-995f-ed8c7e4c7708"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:12:41.268931 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:41.268909 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17f67c08-cc43-45c3-995f-ed8c7e4c7708-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:12:41.270764 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:41.270733 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17f67c08-cc43-45c3-995f-ed8c7e4c7708-maas-api-xks-ca" pod "17f67c08-cc43-45c3-995f-ed8c7e4c7708" (UID: "17f67c08-cc43-45c3-995f-ed8c7e4c7708"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:41.270884 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:41.270790 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f67c08-cc43-45c3-995f-ed8c7e4c7708-kube-api-access-k9vk5" pod "17f67c08-cc43-45c3-995f-ed8c7e4c7708" (UID: "17f67c08-cc43-45c3-995f-ed8c7e4c7708"). InnerVolumeSpecName "kube-api-access-k9vk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:41.370912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:41.370828 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k9vk5\" (UniqueName: \"kubernetes.io/projected/17f67c08-cc43-45c3-995f-ed8c7e4c7708-kube-api-access-k9vk5\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:12:41.370912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:41.370866 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17f67c08-cc43-45c3-995f-ed8c7e4c7708-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:12:42.089649 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:42.089612 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb" event={"ID":"17f67c08-cc43-45c3-995f-ed8c7e4c7708","Type":"ContainerDied","Data":"b80321a22f84d51ea949bbea8e5293ba5d3d48b13a010720a66fbbc8ff414d00"} Oct 30 18:12:42.090009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:42.089992 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b80321a22f84d51ea949bbea8e5293ba5d3d48b13a010720a66fbbc8ff414d00" Oct 30 18:12:42.090144 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:42.089823 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb" Oct 30 18:12:50.133127 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.133094 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh"] Oct 30 18:12:50.133645 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.133599 5232 cpu_manager.go:414] "containerMap: removing container" podUID="17f67c08-cc43-45c3-995f-ed8c7e4c7708" containerName="revoke-keys" Oct 30 18:12:50.133645 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.133610 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f67c08-cc43-45c3-995f-ed8c7e4c7708" containerName="revoke-keys" Oct 30 18:12:50.133732 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.133721 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f67c08-cc43-45c3-995f-ed8c7e4c7708" containerName="revoke-keys" Oct 30 18:12:50.137142 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.137121 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh" Oct 30 18:12:50.164044 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.164011 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:12:50.164176 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.164088 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:12:50.166850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.166828 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh"] Oct 30 18:12:50.226372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.226335 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jw49\" (UniqueName: \"kubernetes.io/projected/546de000-9c73-49b8-8ff4-c090eb595d82-kube-api-access-7jw49\") pod \"maas-api-revoke-sub-70d58ae8-rvtzh\" (UID: \"546de000-9c73-49b8-8ff4-c090eb595d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh" Oct 30 18:12:50.226575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.226533 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546de000-9c73-49b8-8ff4-c090eb595d82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70d58ae8-rvtzh\" (UID: \"546de000-9c73-49b8-8ff4-c090eb595d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh" Oct 30 18:12:50.226689 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.226671 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546de000-9c73-49b8-8ff4-c090eb595d82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70d58ae8-rvtzh\" (UID: \"546de000-9c73-49b8-8ff4-c090eb595d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh" Oct 30 18:12:50.327421 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.327387 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546de000-9c73-49b8-8ff4-c090eb595d82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70d58ae8-rvtzh\" (UID: \"546de000-9c73-49b8-8ff4-c090eb595d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh" Oct 30 18:12:50.327636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.327455 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546de000-9c73-49b8-8ff4-c090eb595d82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70d58ae8-rvtzh\" (UID: \"546de000-9c73-49b8-8ff4-c090eb595d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh" Oct 30 18:12:50.327636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.327481 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jw49\" (UniqueName: \"kubernetes.io/projected/546de000-9c73-49b8-8ff4-c090eb595d82-kube-api-access-7jw49\") pod \"maas-api-revoke-sub-70d58ae8-rvtzh\" (UID: \"546de000-9c73-49b8-8ff4-c090eb595d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh" Oct 30 18:12:50.328119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.328095 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546de000-9c73-49b8-8ff4-c090eb595d82-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-70d58ae8-rvtzh\" (UID: \"546de000-9c73-49b8-8ff4-c090eb595d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh" Oct 30 18:12:50.329980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.329958 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546de000-9c73-49b8-8ff4-c090eb595d82-maas-api-xks-ca\") pod \"maas-api-revoke-sub-70d58ae8-rvtzh\" (UID: \"546de000-9c73-49b8-8ff4-c090eb595d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh" Oct 30 18:12:50.346621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.346594 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jw49\" (UniqueName: \"kubernetes.io/projected/546de000-9c73-49b8-8ff4-c090eb595d82-kube-api-access-7jw49\") pod \"maas-api-revoke-sub-70d58ae8-rvtzh\" (UID: \"546de000-9c73-49b8-8ff4-c090eb595d82\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh" Oct 30 18:12:50.451418 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.451333 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh" Oct 30 18:12:50.745762 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:50.745734 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh"] Oct 30 18:12:50.749077 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:12:50.749050 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546de000_9c73_49b8_8ff4_c090eb595d82.slice/crio-fc17eaa8812c8dd4f61742a81ac4ab883d3757965c10a38a29b2a62ccd7052d4 WatchSource:0}: Error finding container fc17eaa8812c8dd4f61742a81ac4ab883d3757965c10a38a29b2a62ccd7052d4: Status 404 returned error can't find the container with id fc17eaa8812c8dd4f61742a81ac4ab883d3757965c10a38a29b2a62ccd7052d4 Oct 30 18:12:51.148987 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:51.148946 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh" event={"ID":"546de000-9c73-49b8-8ff4-c090eb595d82","Type":"ContainerStarted","Data":"7b1b1b42b07d3e317bd6a206330064e83f81cc8b52cce213bc8f94b1082b0353"} Oct 30 18:12:51.149374 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:51.149007 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh" event={"ID":"546de000-9c73-49b8-8ff4-c090eb595d82","Type":"ContainerStarted","Data":"fc17eaa8812c8dd4f61742a81ac4ab883d3757965c10a38a29b2a62ccd7052d4"} Oct 30 18:12:51.190828 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:51.190779 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh" podStartSLOduration=1.190758162 podStartE2EDuration="1.190758162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:12:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:12:51.188128532 +0000 UTC m=+670.504548853" watchObservedRunningTime="2026-10-30 18:12:51.190758162 +0000 UTC m=+670.507178485" Oct 30 18:12:52.155943 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:52.155905 5232 generic.go:356] "Generic (PLEG): container finished" podID="546de000-9c73-49b8-8ff4-c090eb595d82" containerID="7b1b1b42b07d3e317bd6a206330064e83f81cc8b52cce213bc8f94b1082b0353" exitCode=0 Oct 30 18:12:52.156324 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:52.155984 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh" event={"ID":"546de000-9c73-49b8-8ff4-c090eb595d82","Type":"ContainerDied","Data":"7b1b1b42b07d3e317bd6a206330064e83f81cc8b52cce213bc8f94b1082b0353"} Oct 30 18:12:53.303100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:53.303066 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh" Oct 30 18:12:53.465493 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:53.465414 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/546de000-9c73-49b8-8ff4-c090eb595d82-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546de000-9c73-49b8-8ff4-c090eb595d82-maas-api-openshift-ca\") pod \"546de000-9c73-49b8-8ff4-c090eb595d82\" (UID: \"546de000-9c73-49b8-8ff4-c090eb595d82\") " Oct 30 18:12:53.465493 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:53.465473 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/546de000-9c73-49b8-8ff4-c090eb595d82-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546de000-9c73-49b8-8ff4-c090eb595d82-maas-api-xks-ca\") pod \"546de000-9c73-49b8-8ff4-c090eb595d82\" (UID: \"546de000-9c73-49b8-8ff4-c090eb595d82\") " Oct 30 18:12:53.465697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:53.465556 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/546de000-9c73-49b8-8ff4-c090eb595d82-kube-api-access-7jw49\" (UniqueName: \"kubernetes.io/projected/546de000-9c73-49b8-8ff4-c090eb595d82-kube-api-access-7jw49\") pod \"546de000-9c73-49b8-8ff4-c090eb595d82\" (UID: \"546de000-9c73-49b8-8ff4-c090eb595d82\") " Oct 30 18:12:53.465898 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:53.465864 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/546de000-9c73-49b8-8ff4-c090eb595d82-maas-api-openshift-ca" pod "546de000-9c73-49b8-8ff4-c090eb595d82" (UID: "546de000-9c73-49b8-8ff4-c090eb595d82"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:12:53.466024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:53.466007 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546de000-9c73-49b8-8ff4-c090eb595d82-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:12:53.467778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:53.467752 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/546de000-9c73-49b8-8ff4-c090eb595d82-maas-api-xks-ca" pod "546de000-9c73-49b8-8ff4-c090eb595d82" (UID: "546de000-9c73-49b8-8ff4-c090eb595d82"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:12:53.467866 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:53.467771 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/546de000-9c73-49b8-8ff4-c090eb595d82-kube-api-access-7jw49" pod "546de000-9c73-49b8-8ff4-c090eb595d82" (UID: "546de000-9c73-49b8-8ff4-c090eb595d82"). InnerVolumeSpecName "kube-api-access-7jw49". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:12:53.567010 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:53.566979 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546de000-9c73-49b8-8ff4-c090eb595d82-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:12:53.567010 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:53.567007 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jw49\" (UniqueName: \"kubernetes.io/projected/546de000-9c73-49b8-8ff4-c090eb595d82-kube-api-access-7jw49\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:12:54.167934 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:54.167906 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh" Oct 30 18:12:54.168112 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:54.167924 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh" event={"ID":"546de000-9c73-49b8-8ff4-c090eb595d82","Type":"ContainerDied","Data":"fc17eaa8812c8dd4f61742a81ac4ab883d3757965c10a38a29b2a62ccd7052d4"} Oct 30 18:12:54.168112 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:54.167982 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fc17eaa8812c8dd4f61742a81ac4ab883d3757965c10a38a29b2a62ccd7052d4" Oct 30 18:12:57.033803 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:12:57.033768 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:12:58.493628 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.493598 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz"] Oct 30 18:12:58.494155 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.494143 5232 cpu_manager.go:414] "containerMap: removing container" podUID="546de000-9c73-49b8-8ff4-c090eb595d82" containerName="revoke-keys" Oct 30 18:12:58.494213 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.494156 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="546de000-9c73-49b8-8ff4-c090eb595d82" containerName="revoke-keys" Oct 30 18:12:58.494287 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.494277 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="546de000-9c73-49b8-8ff4-c090eb595d82" containerName="revoke-keys" Oct 30 18:12:58.497887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.497863 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz" Oct 30 18:12:58.516972 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.516946 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz"] Oct 30 18:12:58.524639 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.524573 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:12:58.524639 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.524642 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:12:58.570452 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.570413 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m"] Oct 30 18:12:58.574573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.574550 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m" Oct 30 18:12:58.591960 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.591934 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m"] Oct 30 18:12:58.638873 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.638841 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86gwr\" (UniqueName: \"kubernetes.io/projected/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-kube-api-access-86gwr\") pod \"maas-api-revoke-sub-7421d87b-hb5rz\" (UID: \"757d4a71-96c1-4d34-a8a4-2e0cf6328e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz" Oct 30 18:12:58.639060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.639034 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7421d87b-hb5rz\" (UID: \"757d4a71-96c1-4d34-a8a4-2e0cf6328e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz" Oct 30 18:12:58.639128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.639092 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7421d87b-hb5rz\" (UID: \"757d4a71-96c1-4d34-a8a4-2e0cf6328e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz" Oct 30 18:12:58.740206 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.740163 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86gwr\" (UniqueName: \"kubernetes.io/projected/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-kube-api-access-86gwr\") pod \"maas-api-revoke-sub-7421d87b-hb5rz\" (UID: \"757d4a71-96c1-4d34-a8a4-2e0cf6328e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz" Oct 30 18:12:58.740383 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.740248 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a77ef0ef-32af-46d9-86e5-8fc577144050-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57a50f8c-f2m5m\" (UID: \"a77ef0ef-32af-46d9-86e5-8fc577144050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m" Oct 30 18:12:58.740383 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.740332 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a77ef0ef-32af-46d9-86e5-8fc577144050-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57a50f8c-f2m5m\" (UID: \"a77ef0ef-32af-46d9-86e5-8fc577144050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m" Oct 30 18:12:58.740529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.740458 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7421d87b-hb5rz\" (UID: \"757d4a71-96c1-4d34-a8a4-2e0cf6328e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz" Oct 30 18:12:58.740589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.740552 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7421d87b-hb5rz\" (UID: \"757d4a71-96c1-4d34-a8a4-2e0cf6328e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz" Oct 30 18:12:58.740643 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.740626 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5x78b\" (UniqueName: \"kubernetes.io/projected/a77ef0ef-32af-46d9-86e5-8fc577144050-kube-api-access-5x78b\") pod \"maas-api-revoke-sub-57a50f8c-f2m5m\" (UID: \"a77ef0ef-32af-46d9-86e5-8fc577144050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m" Oct 30 18:12:58.741156 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.741136 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7421d87b-hb5rz\" (UID: \"757d4a71-96c1-4d34-a8a4-2e0cf6328e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz" Oct 30 18:12:58.743081 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.743061 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7421d87b-hb5rz\" (UID: \"757d4a71-96c1-4d34-a8a4-2e0cf6328e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz" Oct 30 18:12:58.759461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.759400 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86gwr\" (UniqueName: \"kubernetes.io/projected/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-kube-api-access-86gwr\") pod \"maas-api-revoke-sub-7421d87b-hb5rz\" (UID: \"757d4a71-96c1-4d34-a8a4-2e0cf6328e18\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz" Oct 30 18:12:58.812895 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.812866 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz" Oct 30 18:12:58.842190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.842153 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5x78b\" (UniqueName: \"kubernetes.io/projected/a77ef0ef-32af-46d9-86e5-8fc577144050-kube-api-access-5x78b\") pod \"maas-api-revoke-sub-57a50f8c-f2m5m\" (UID: \"a77ef0ef-32af-46d9-86e5-8fc577144050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m" Oct 30 18:12:58.842341 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.842279 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a77ef0ef-32af-46d9-86e5-8fc577144050-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57a50f8c-f2m5m\" (UID: \"a77ef0ef-32af-46d9-86e5-8fc577144050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m" Oct 30 18:12:58.842411 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.842357 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a77ef0ef-32af-46d9-86e5-8fc577144050-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57a50f8c-f2m5m\" (UID: \"a77ef0ef-32af-46d9-86e5-8fc577144050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m" Oct 30 18:12:58.843024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.843002 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a77ef0ef-32af-46d9-86e5-8fc577144050-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-57a50f8c-f2m5m\" (UID: \"a77ef0ef-32af-46d9-86e5-8fc577144050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m" Oct 30 18:12:58.845326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.845298 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a77ef0ef-32af-46d9-86e5-8fc577144050-maas-api-xks-ca\") pod \"maas-api-revoke-sub-57a50f8c-f2m5m\" (UID: \"a77ef0ef-32af-46d9-86e5-8fc577144050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m" Oct 30 18:12:58.861693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.861667 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5x78b\" (UniqueName: \"kubernetes.io/projected/a77ef0ef-32af-46d9-86e5-8fc577144050-kube-api-access-5x78b\") pod \"maas-api-revoke-sub-57a50f8c-f2m5m\" (UID: \"a77ef0ef-32af-46d9-86e5-8fc577144050\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m" Oct 30 18:12:58.887895 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:58.887862 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m" Oct 30 18:12:59.115052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:59.114763 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz"] Oct 30 18:12:59.119139 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:12:59.119112 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod757d4a71_96c1_4d34_a8a4_2e0cf6328e18.slice/crio-9748b601bbbebd5c81c6f39a4d70570600a4307d9cca963c7d9bb1c7b44080ba WatchSource:0}: Error finding container 9748b601bbbebd5c81c6f39a4d70570600a4307d9cca963c7d9bb1c7b44080ba: Status 404 returned error can't find the container with id 9748b601bbbebd5c81c6f39a4d70570600a4307d9cca963c7d9bb1c7b44080ba Oct 30 18:12:59.194994 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:59.194967 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m"] Oct 30 18:12:59.198871 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:12:59.198843 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz" event={"ID":"757d4a71-96c1-4d34-a8a4-2e0cf6328e18","Type":"ContainerStarted","Data":"9748b601bbbebd5c81c6f39a4d70570600a4307d9cca963c7d9bb1c7b44080ba"} Oct 30 18:12:59.199709 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:12:59.199687 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda77ef0ef_32af_46d9_86e5_8fc577144050.slice/crio-fae41fd2b38041fbd0af0f786e15721073a79c0a47a84c0e5b9e3308932db889 WatchSource:0}: Error finding container fae41fd2b38041fbd0af0f786e15721073a79c0a47a84c0e5b9e3308932db889: Status 404 returned error can't find the container with id fae41fd2b38041fbd0af0f786e15721073a79c0a47a84c0e5b9e3308932db889 Oct 30 18:13:00.206006 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:00.205973 5232 generic.go:356] "Generic (PLEG): container finished" podID="757d4a71-96c1-4d34-a8a4-2e0cf6328e18" containerID="3c182b65baf88209cab82f401db76605162ac7bc4c885c18580438afdd1a7f8d" exitCode=0 Oct 30 18:13:00.206451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:00.206063 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz" event={"ID":"757d4a71-96c1-4d34-a8a4-2e0cf6328e18","Type":"ContainerDied","Data":"3c182b65baf88209cab82f401db76605162ac7bc4c885c18580438afdd1a7f8d"} Oct 30 18:13:00.207678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:00.207569 5232 generic.go:356] "Generic (PLEG): container finished" podID="a77ef0ef-32af-46d9-86e5-8fc577144050" containerID="fdffb323bd05e3cdf5847972c2838c829b94871990cb9cad125a56e6dcf8d952" exitCode=0 Oct 30 18:13:00.207678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:00.207647 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m" event={"ID":"a77ef0ef-32af-46d9-86e5-8fc577144050","Type":"ContainerDied","Data":"fdffb323bd05e3cdf5847972c2838c829b94871990cb9cad125a56e6dcf8d952"} Oct 30 18:13:00.207678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:00.207677 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m" event={"ID":"a77ef0ef-32af-46d9-86e5-8fc577144050","Type":"ContainerStarted","Data":"fae41fd2b38041fbd0af0f786e15721073a79c0a47a84c0e5b9e3308932db889"} Oct 30 18:13:01.384988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.384966 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m" Oct 30 18:13:01.387705 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.387684 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz" Oct 30 18:13:01.472377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.472287 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a77ef0ef-32af-46d9-86e5-8fc577144050-kube-api-access-5x78b\" (UniqueName: \"kubernetes.io/projected/a77ef0ef-32af-46d9-86e5-8fc577144050-kube-api-access-5x78b\") pod \"a77ef0ef-32af-46d9-86e5-8fc577144050\" (UID: \"a77ef0ef-32af-46d9-86e5-8fc577144050\") " Oct 30 18:13:01.472556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.472415 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a77ef0ef-32af-46d9-86e5-8fc577144050-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a77ef0ef-32af-46d9-86e5-8fc577144050-maas-api-xks-ca\") pod \"a77ef0ef-32af-46d9-86e5-8fc577144050\" (UID: \"a77ef0ef-32af-46d9-86e5-8fc577144050\") " Oct 30 18:13:01.472626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.472576 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a77ef0ef-32af-46d9-86e5-8fc577144050-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a77ef0ef-32af-46d9-86e5-8fc577144050-maas-api-openshift-ca\") pod \"a77ef0ef-32af-46d9-86e5-8fc577144050\" (UID: \"a77ef0ef-32af-46d9-86e5-8fc577144050\") " Oct 30 18:13:01.473766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.473717 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a77ef0ef-32af-46d9-86e5-8fc577144050-maas-api-openshift-ca" pod "a77ef0ef-32af-46d9-86e5-8fc577144050" (UID: "a77ef0ef-32af-46d9-86e5-8fc577144050"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:01.482022 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.481986 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a77ef0ef-32af-46d9-86e5-8fc577144050-maas-api-xks-ca" pod "a77ef0ef-32af-46d9-86e5-8fc577144050" (UID: "a77ef0ef-32af-46d9-86e5-8fc577144050"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:01.482176 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.482068 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a77ef0ef-32af-46d9-86e5-8fc577144050-kube-api-access-5x78b" pod "a77ef0ef-32af-46d9-86e5-8fc577144050" (UID: "a77ef0ef-32af-46d9-86e5-8fc577144050"). InnerVolumeSpecName "kube-api-access-5x78b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:01.573660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.573631 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-maas-api-xks-ca\") pod \"757d4a71-96c1-4d34-a8a4-2e0cf6328e18\" (UID: \"757d4a71-96c1-4d34-a8a4-2e0cf6328e18\") " Oct 30 18:13:01.573860 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.573798 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-kube-api-access-86gwr\" (UniqueName: \"kubernetes.io/projected/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-kube-api-access-86gwr\") pod \"757d4a71-96c1-4d34-a8a4-2e0cf6328e18\" (UID: \"757d4a71-96c1-4d34-a8a4-2e0cf6328e18\") " Oct 30 18:13:01.573860 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.573829 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-maas-api-openshift-ca\") pod \"757d4a71-96c1-4d34-a8a4-2e0cf6328e18\" (UID: \"757d4a71-96c1-4d34-a8a4-2e0cf6328e18\") " Oct 30 18:13:01.574285 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.574253 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-maas-api-openshift-ca" pod "757d4a71-96c1-4d34-a8a4-2e0cf6328e18" (UID: "757d4a71-96c1-4d34-a8a4-2e0cf6328e18"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:01.574285 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.574270 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a77ef0ef-32af-46d9-86e5-8fc577144050-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:01.574447 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.574294 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a77ef0ef-32af-46d9-86e5-8fc577144050-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:01.574447 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.574307 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5x78b\" (UniqueName: \"kubernetes.io/projected/a77ef0ef-32af-46d9-86e5-8fc577144050-kube-api-access-5x78b\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:01.575988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.575961 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-maas-api-xks-ca" pod "757d4a71-96c1-4d34-a8a4-2e0cf6328e18" (UID: "757d4a71-96c1-4d34-a8a4-2e0cf6328e18"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:01.576150 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.576135 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-kube-api-access-86gwr" pod "757d4a71-96c1-4d34-a8a4-2e0cf6328e18" (UID: "757d4a71-96c1-4d34-a8a4-2e0cf6328e18"). InnerVolumeSpecName "kube-api-access-86gwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:01.675119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.675078 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:01.675119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.675112 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86gwr\" (UniqueName: \"kubernetes.io/projected/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-kube-api-access-86gwr\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:01.675119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:01.675123 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/757d4a71-96c1-4d34-a8a4-2e0cf6328e18-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:02.219279 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.219243 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz" Oct 30 18:13:02.219479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.219290 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz" event={"ID":"757d4a71-96c1-4d34-a8a4-2e0cf6328e18","Type":"ContainerDied","Data":"9748b601bbbebd5c81c6f39a4d70570600a4307d9cca963c7d9bb1c7b44080ba"} Oct 30 18:13:02.219479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.219351 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9748b601bbbebd5c81c6f39a4d70570600a4307d9cca963c7d9bb1c7b44080ba" Oct 30 18:13:02.220811 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.220788 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m" event={"ID":"a77ef0ef-32af-46d9-86e5-8fc577144050","Type":"ContainerDied","Data":"fae41fd2b38041fbd0af0f786e15721073a79c0a47a84c0e5b9e3308932db889"} Oct 30 18:13:02.220942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.220813 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m" Oct 30 18:13:02.220942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.220822 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fae41fd2b38041fbd0af0f786e15721073a79c0a47a84c0e5b9e3308932db889" Oct 30 18:13:02.629686 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.629656 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww"] Oct 30 18:13:02.630345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.630323 5232 cpu_manager.go:414] "containerMap: removing container" podUID="a77ef0ef-32af-46d9-86e5-8fc577144050" containerName="revoke-keys" Oct 30 18:13:02.630434 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.630347 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a77ef0ef-32af-46d9-86e5-8fc577144050" containerName="revoke-keys" Oct 30 18:13:02.630434 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.630385 5232 cpu_manager.go:414] "containerMap: removing container" podUID="757d4a71-96c1-4d34-a8a4-2e0cf6328e18" containerName="revoke-keys" Oct 30 18:13:02.630434 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.630395 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="757d4a71-96c1-4d34-a8a4-2e0cf6328e18" containerName="revoke-keys" Oct 30 18:13:02.630665 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.630544 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="a77ef0ef-32af-46d9-86e5-8fc577144050" containerName="revoke-keys" Oct 30 18:13:02.630665 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.630563 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="757d4a71-96c1-4d34-a8a4-2e0cf6328e18" containerName="revoke-keys" Oct 30 18:13:02.634039 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.634021 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww" Oct 30 18:13:02.649166 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.649144 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:13:02.651254 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.651229 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww"] Oct 30 18:13:02.660724 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.660700 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:13:02.790524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.790475 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89fhh\" (UniqueName: \"kubernetes.io/projected/e06f351d-6504-44b5-a76b-90d9eaff99b2-kube-api-access-89fhh\") pod \"maas-api-revoke-sub-383e3cd8-trkww\" (UID: \"e06f351d-6504-44b5-a76b-90d9eaff99b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww" Oct 30 18:13:02.790694 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.790640 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06f351d-6504-44b5-a76b-90d9eaff99b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-383e3cd8-trkww\" (UID: \"e06f351d-6504-44b5-a76b-90d9eaff99b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww" Oct 30 18:13:02.790694 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.790686 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06f351d-6504-44b5-a76b-90d9eaff99b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-383e3cd8-trkww\" (UID: \"e06f351d-6504-44b5-a76b-90d9eaff99b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww" Oct 30 18:13:02.892212 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.892124 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06f351d-6504-44b5-a76b-90d9eaff99b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-383e3cd8-trkww\" (UID: \"e06f351d-6504-44b5-a76b-90d9eaff99b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww" Oct 30 18:13:02.892212 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.892184 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06f351d-6504-44b5-a76b-90d9eaff99b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-383e3cd8-trkww\" (UID: \"e06f351d-6504-44b5-a76b-90d9eaff99b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww" Oct 30 18:13:02.892418 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.892330 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89fhh\" (UniqueName: \"kubernetes.io/projected/e06f351d-6504-44b5-a76b-90d9eaff99b2-kube-api-access-89fhh\") pod \"maas-api-revoke-sub-383e3cd8-trkww\" (UID: \"e06f351d-6504-44b5-a76b-90d9eaff99b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww" Oct 30 18:13:02.893075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.892986 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06f351d-6504-44b5-a76b-90d9eaff99b2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-383e3cd8-trkww\" (UID: \"e06f351d-6504-44b5-a76b-90d9eaff99b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww" Oct 30 18:13:02.894734 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.894705 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06f351d-6504-44b5-a76b-90d9eaff99b2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-383e3cd8-trkww\" (UID: \"e06f351d-6504-44b5-a76b-90d9eaff99b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww" Oct 30 18:13:02.911388 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.911363 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89fhh\" (UniqueName: \"kubernetes.io/projected/e06f351d-6504-44b5-a76b-90d9eaff99b2-kube-api-access-89fhh\") pod \"maas-api-revoke-sub-383e3cd8-trkww\" (UID: \"e06f351d-6504-44b5-a76b-90d9eaff99b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww" Oct 30 18:13:02.948052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:02.948011 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww" Oct 30 18:13:03.241465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:03.241429 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww"] Oct 30 18:13:03.244568 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:13:03.244536 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode06f351d_6504_44b5_a76b_90d9eaff99b2.slice/crio-0eaaf8cd6ffca5f29814850524239d688088b66d7ca85b666f210035a6c9214f WatchSource:0}: Error finding container 0eaaf8cd6ffca5f29814850524239d688088b66d7ca85b666f210035a6c9214f: Status 404 returned error can't find the container with id 0eaaf8cd6ffca5f29814850524239d688088b66d7ca85b666f210035a6c9214f Oct 30 18:13:04.233232 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:04.233201 5232 generic.go:356] "Generic (PLEG): container finished" podID="e06f351d-6504-44b5-a76b-90d9eaff99b2" containerID="2217292207ec98fb25c87a4f740ecae77cc128cd2c9b9136884c54e2fda78862" exitCode=0 Oct 30 18:13:04.233648 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:04.233235 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww" event={"ID":"e06f351d-6504-44b5-a76b-90d9eaff99b2","Type":"ContainerDied","Data":"2217292207ec98fb25c87a4f740ecae77cc128cd2c9b9136884c54e2fda78862"} Oct 30 18:13:04.233648 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:04.233289 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww" event={"ID":"e06f351d-6504-44b5-a76b-90d9eaff99b2","Type":"ContainerStarted","Data":"0eaaf8cd6ffca5f29814850524239d688088b66d7ca85b666f210035a6c9214f"} Oct 30 18:13:05.391002 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:05.390975 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww" Oct 30 18:13:05.425797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:05.425765 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e06f351d-6504-44b5-a76b-90d9eaff99b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06f351d-6504-44b5-a76b-90d9eaff99b2-maas-api-xks-ca\") pod \"e06f351d-6504-44b5-a76b-90d9eaff99b2\" (UID: \"e06f351d-6504-44b5-a76b-90d9eaff99b2\") " Oct 30 18:13:05.425960 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:05.425820 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e06f351d-6504-44b5-a76b-90d9eaff99b2-kube-api-access-89fhh\" (UniqueName: \"kubernetes.io/projected/e06f351d-6504-44b5-a76b-90d9eaff99b2-kube-api-access-89fhh\") pod \"e06f351d-6504-44b5-a76b-90d9eaff99b2\" (UID: \"e06f351d-6504-44b5-a76b-90d9eaff99b2\") " Oct 30 18:13:05.425960 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:05.425916 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e06f351d-6504-44b5-a76b-90d9eaff99b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06f351d-6504-44b5-a76b-90d9eaff99b2-maas-api-openshift-ca\") pod \"e06f351d-6504-44b5-a76b-90d9eaff99b2\" (UID: \"e06f351d-6504-44b5-a76b-90d9eaff99b2\") " Oct 30 18:13:05.426273 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:05.426250 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e06f351d-6504-44b5-a76b-90d9eaff99b2-maas-api-openshift-ca" pod "e06f351d-6504-44b5-a76b-90d9eaff99b2" (UID: "e06f351d-6504-44b5-a76b-90d9eaff99b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:05.428855 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:05.428824 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e06f351d-6504-44b5-a76b-90d9eaff99b2-maas-api-xks-ca" pod "e06f351d-6504-44b5-a76b-90d9eaff99b2" (UID: "e06f351d-6504-44b5-a76b-90d9eaff99b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:05.429244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:05.429211 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e06f351d-6504-44b5-a76b-90d9eaff99b2-kube-api-access-89fhh" pod "e06f351d-6504-44b5-a76b-90d9eaff99b2" (UID: "e06f351d-6504-44b5-a76b-90d9eaff99b2"). InnerVolumeSpecName "kube-api-access-89fhh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:05.527024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:05.526934 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e06f351d-6504-44b5-a76b-90d9eaff99b2-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:05.527024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:05.526962 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89fhh\" (UniqueName: \"kubernetes.io/projected/e06f351d-6504-44b5-a76b-90d9eaff99b2-kube-api-access-89fhh\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:05.527024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:05.526971 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e06f351d-6504-44b5-a76b-90d9eaff99b2-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:06.245990 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:06.245958 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww" Oct 30 18:13:06.246170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:06.245994 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww" event={"ID":"e06f351d-6504-44b5-a76b-90d9eaff99b2","Type":"ContainerDied","Data":"0eaaf8cd6ffca5f29814850524239d688088b66d7ca85b666f210035a6c9214f"} Oct 30 18:13:06.246170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:06.246041 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eaaf8cd6ffca5f29814850524239d688088b66d7ca85b666f210035a6c9214f" Oct 30 18:13:09.219006 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.218968 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf"] Oct 30 18:13:09.219464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.219450 5232 cpu_manager.go:414] "containerMap: removing container" podUID="e06f351d-6504-44b5-a76b-90d9eaff99b2" containerName="revoke-keys" Oct 30 18:13:09.219464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.219465 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e06f351d-6504-44b5-a76b-90d9eaff99b2" containerName="revoke-keys" Oct 30 18:13:09.219629 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.219615 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="e06f351d-6504-44b5-a76b-90d9eaff99b2" containerName="revoke-keys" Oct 30 18:13:09.222986 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.222965 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.252976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.252934 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf"] Oct 30 18:13:09.288071 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.288039 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9364c01-b339-4517-b6d7-290895898b00-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.288246 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.288078 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9364c01-b339-4517-b6d7-290895898b00-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.288333 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.288244 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.288333 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.288287 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rqss\" (UniqueName: \"kubernetes.io/projected/b9364c01-b339-4517-b6d7-290895898b00-kube-api-access-4rqss\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.288472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.288444 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.288596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.288537 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.288653 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.288591 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9364c01-b339-4517-b6d7-290895898b00-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.288717 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.288699 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.288865 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.288841 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.389949 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.389912 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.390134 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.389963 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.390134 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.389980 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9364c01-b339-4517-b6d7-290895898b00-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.390134 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.390043 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.390134 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.390083 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.390134 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.390112 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9364c01-b339-4517-b6d7-290895898b00-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.390395 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.390140 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9364c01-b339-4517-b6d7-290895898b00-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.390395 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.390255 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.390395 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.390289 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rqss\" (UniqueName: \"kubernetes.io/projected/b9364c01-b339-4517-b6d7-290895898b00-kube-api-access-4rqss\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.390602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.390450 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-workload-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.390836 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.390811 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-workload-certs\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.390920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.390846 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9364c01-b339-4517-b6d7-290895898b00-istiod-ca-cert\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.390920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.390865 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-credential-socket\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.390994 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.390918 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-istio-data\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.392499 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.392477 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-istio-envoy\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.392846 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.392827 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9364c01-b339-4517-b6d7-290895898b00-istio-podinfo\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.410712 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.410687 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9364c01-b339-4517-b6d7-290895898b00-istio-token\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.411115 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.411098 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rqss\" (UniqueName: \"kubernetes.io/projected/b9364c01-b339-4517-b6d7-290895898b00-kube-api-access-4rqss\") pod \"e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.537235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.537196 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:09.894221 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:13:09.893836 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9364c01_b339_4517_b6d7_290895898b00.slice/crio-604089073d0f795d0a93ec134750b13de21024ef3281a30796c6337247ff0378 WatchSource:0}: Error finding container 604089073d0f795d0a93ec134750b13de21024ef3281a30796c6337247ff0378: Status 404 returned error can't find the container with id 604089073d0f795d0a93ec134750b13de21024ef3281a30796c6337247ff0378 Oct 30 18:13:09.894914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.894885 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf"] Oct 30 18:13:09.902536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.899382 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:13:09.902536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.899483 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:13:09.902536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:09.899554 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:13:10.277980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:10.277929 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" event={"ID":"b9364c01-b339-4517-b6d7-290895898b00","Type":"ContainerStarted","Data":"ddfbd1488ed0e7c097476de2d389ce8441e068549acce256e1e37729a2f558f3"} Oct 30 18:13:10.278473 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:10.278004 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" event={"ID":"b9364c01-b339-4517-b6d7-290895898b00","Type":"ContainerStarted","Data":"604089073d0f795d0a93ec134750b13de21024ef3281a30796c6337247ff0378"} Oct 30 18:13:10.320291 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:10.320230 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" podStartSLOduration=1.3202122840000001 podStartE2EDuration="1.320212284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:10.318630434 +0000 UTC m=+689.635050779" watchObservedRunningTime="2026-10-30 18:13:10.320212284 +0000 UTC m=+689.636632609" Oct 30 18:13:10.537823 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:10.537699 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:11.539535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:11.539401 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:13:11.539535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:11.539464 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" podUID="b9364c01-b339-4517-b6d7-290895898b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:13:12.537822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:12.537772 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:13:12.538012 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:12.537840 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" podUID="b9364c01-b339-4517-b6d7-290895898b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:13:13.537764 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:13.537723 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:13:13.538200 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:13.537790 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" podUID="b9364c01-b339-4517-b6d7-290895898b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:13:14.538245 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:14.538207 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:13:14.538703 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:14.538276 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" podUID="b9364c01-b339-4517-b6d7-290895898b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:13:15.538527 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:15.538477 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:13:15.539025 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:15.538565 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" podUID="b9364c01-b339-4517-b6d7-290895898b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:13:16.538474 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:16.538430 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" start-of-body= Oct 30 18:13:16.538695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:16.538482 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" podUID="b9364c01-b339-4517-b6d7-290895898b00" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.58:15021/healthz/ready\": dial tcp 10.133.0.58:15021: connect: connection refused" Oct 30 18:13:17.542653 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:17.542480 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:17.543119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:17.543033 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:17.543865 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:17.543844 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:13:20.598004 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:20.597970 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm"] Oct 30 18:13:20.605477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:20.605455 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm" Oct 30 18:13:20.619523 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:20.619483 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm"] Oct 30 18:13:20.627777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:20.627751 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:13:20.627953 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:20.627844 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:13:20.657908 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:20.657880 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed26c08a-qkrsm\" (UID: \"c2322f42-c68c-4e5a-a7ae-ed6b204728b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm" Oct 30 18:13:20.658065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:20.657919 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zkrj\" (UniqueName: \"kubernetes.io/projected/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-kube-api-access-2zkrj\") pod \"maas-api-revoke-sub-ed26c08a-qkrsm\" (UID: \"c2322f42-c68c-4e5a-a7ae-ed6b204728b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm" Oct 30 18:13:20.658107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:20.658084 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed26c08a-qkrsm\" (UID: \"c2322f42-c68c-4e5a-a7ae-ed6b204728b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm" Oct 30 18:13:20.759492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:20.759456 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed26c08a-qkrsm\" (UID: \"c2322f42-c68c-4e5a-a7ae-ed6b204728b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm" Oct 30 18:13:20.759681 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:20.759504 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zkrj\" (UniqueName: \"kubernetes.io/projected/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-kube-api-access-2zkrj\") pod \"maas-api-revoke-sub-ed26c08a-qkrsm\" (UID: \"c2322f42-c68c-4e5a-a7ae-ed6b204728b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm" Oct 30 18:13:20.759681 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:20.759635 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed26c08a-qkrsm\" (UID: \"c2322f42-c68c-4e5a-a7ae-ed6b204728b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm" Oct 30 18:13:20.760277 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:20.760252 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed26c08a-qkrsm\" (UID: \"c2322f42-c68c-4e5a-a7ae-ed6b204728b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm" Oct 30 18:13:20.762033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:20.762014 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed26c08a-qkrsm\" (UID: \"c2322f42-c68c-4e5a-a7ae-ed6b204728b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm" Oct 30 18:13:20.778325 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:20.778304 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zkrj\" (UniqueName: \"kubernetes.io/projected/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-kube-api-access-2zkrj\") pod \"maas-api-revoke-sub-ed26c08a-qkrsm\" (UID: \"c2322f42-c68c-4e5a-a7ae-ed6b204728b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm" Oct 30 18:13:20.920873 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:20.920799 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm" Oct 30 18:13:21.217244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:21.217163 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm"] Oct 30 18:13:21.221603 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:13:21.221568 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2322f42_c68c_4e5a_a7ae_ed6b204728b6.slice/crio-ea411eb44f8c10c46790f75f8b538d5bd1ab30255eb8e2d5a057010ef4655631 WatchSource:0}: Error finding container ea411eb44f8c10c46790f75f8b538d5bd1ab30255eb8e2d5a057010ef4655631: Status 404 returned error can't find the container with id ea411eb44f8c10c46790f75f8b538d5bd1ab30255eb8e2d5a057010ef4655631 Oct 30 18:13:21.381791 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:21.381757 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm" event={"ID":"c2322f42-c68c-4e5a-a7ae-ed6b204728b6","Type":"ContainerStarted","Data":"ea411eb44f8c10c46790f75f8b538d5bd1ab30255eb8e2d5a057010ef4655631"} Oct 30 18:13:22.388942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:22.388859 5232 generic.go:356] "Generic (PLEG): container finished" podID="c2322f42-c68c-4e5a-a7ae-ed6b204728b6" containerID="1cf92e3b9e0067347d91f3010722fd5c475cecc314cb9909f6e32dea97744d46" exitCode=0 Oct 30 18:13:22.388942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:22.388914 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm" event={"ID":"c2322f42-c68c-4e5a-a7ae-ed6b204728b6","Type":"ContainerDied","Data":"1cf92e3b9e0067347d91f3010722fd5c475cecc314cb9909f6e32dea97744d46"} Oct 30 18:13:23.554739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:23.554705 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm" Oct 30 18:13:23.698109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:23.698021 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-maas-api-xks-ca\") pod \"c2322f42-c68c-4e5a-a7ae-ed6b204728b6\" (UID: \"c2322f42-c68c-4e5a-a7ae-ed6b204728b6\") " Oct 30 18:13:23.698476 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:23.698119 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-kube-api-access-2zkrj\" (UniqueName: \"kubernetes.io/projected/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-kube-api-access-2zkrj\") pod \"c2322f42-c68c-4e5a-a7ae-ed6b204728b6\" (UID: \"c2322f42-c68c-4e5a-a7ae-ed6b204728b6\") " Oct 30 18:13:23.698476 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:23.698153 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-maas-api-openshift-ca\") pod \"c2322f42-c68c-4e5a-a7ae-ed6b204728b6\" (UID: \"c2322f42-c68c-4e5a-a7ae-ed6b204728b6\") " Oct 30 18:13:23.699258 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:23.699222 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-maas-api-openshift-ca" pod "c2322f42-c68c-4e5a-a7ae-ed6b204728b6" (UID: "c2322f42-c68c-4e5a-a7ae-ed6b204728b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:23.700417 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:23.700391 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-maas-api-xks-ca" pod "c2322f42-c68c-4e5a-a7ae-ed6b204728b6" (UID: "c2322f42-c68c-4e5a-a7ae-ed6b204728b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:23.700851 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:23.700825 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-kube-api-access-2zkrj" pod "c2322f42-c68c-4e5a-a7ae-ed6b204728b6" (UID: "c2322f42-c68c-4e5a-a7ae-ed6b204728b6"). InnerVolumeSpecName "kube-api-access-2zkrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:23.819493 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:23.809300 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:23.819893 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:23.819865 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zkrj\" (UniqueName: \"kubernetes.io/projected/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-kube-api-access-2zkrj\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:23.820052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:23.820035 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2322f42-c68c-4e5a-a7ae-ed6b204728b6-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:24.401594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:24.401564 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm" Oct 30 18:13:24.401743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:24.401606 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm" event={"ID":"c2322f42-c68c-4e5a-a7ae-ed6b204728b6","Type":"ContainerDied","Data":"ea411eb44f8c10c46790f75f8b538d5bd1ab30255eb8e2d5a057010ef4655631"} Oct 30 18:13:24.401743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:24.401653 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea411eb44f8c10c46790f75f8b538d5bd1ab30255eb8e2d5a057010ef4655631" Oct 30 18:13:25.693602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:25.693567 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk"] Oct 30 18:13:25.694090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:25.694077 5232 cpu_manager.go:414] "containerMap: removing container" podUID="c2322f42-c68c-4e5a-a7ae-ed6b204728b6" containerName="revoke-keys" Oct 30 18:13:25.694150 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:25.694091 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2322f42-c68c-4e5a-a7ae-ed6b204728b6" containerName="revoke-keys" Oct 30 18:13:25.694187 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:25.694168 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2322f42-c68c-4e5a-a7ae-ed6b204728b6" containerName="revoke-keys" Oct 30 18:13:25.697555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:25.697534 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk" Oct 30 18:13:25.714743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:25.714712 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk"] Oct 30 18:13:25.723111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:25.723068 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:13:25.723291 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:25.723158 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:13:25.842963 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:25.842926 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5a28e0-197e-430e-996e-38f50ab0b723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-130ee7b3-tjxzk\" (UID: \"9d5a28e0-197e-430e-996e-38f50ab0b723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk" Oct 30 18:13:25.843148 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:25.843122 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kpg25\" (UniqueName: \"kubernetes.io/projected/9d5a28e0-197e-430e-996e-38f50ab0b723-kube-api-access-kpg25\") pod \"maas-api-revoke-sub-130ee7b3-tjxzk\" (UID: \"9d5a28e0-197e-430e-996e-38f50ab0b723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk" Oct 30 18:13:25.843209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:25.843181 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5a28e0-197e-430e-996e-38f50ab0b723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-130ee7b3-tjxzk\" (UID: \"9d5a28e0-197e-430e-996e-38f50ab0b723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk" Oct 30 18:13:25.944589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:25.944473 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5a28e0-197e-430e-996e-38f50ab0b723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-130ee7b3-tjxzk\" (UID: \"9d5a28e0-197e-430e-996e-38f50ab0b723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk" Oct 30 18:13:25.944776 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:25.944733 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kpg25\" (UniqueName: \"kubernetes.io/projected/9d5a28e0-197e-430e-996e-38f50ab0b723-kube-api-access-kpg25\") pod \"maas-api-revoke-sub-130ee7b3-tjxzk\" (UID: \"9d5a28e0-197e-430e-996e-38f50ab0b723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk" Oct 30 18:13:25.944842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:25.944814 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5a28e0-197e-430e-996e-38f50ab0b723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-130ee7b3-tjxzk\" (UID: \"9d5a28e0-197e-430e-996e-38f50ab0b723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk" Oct 30 18:13:25.945478 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:25.945449 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5a28e0-197e-430e-996e-38f50ab0b723-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-130ee7b3-tjxzk\" (UID: \"9d5a28e0-197e-430e-996e-38f50ab0b723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk" Oct 30 18:13:25.947190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:25.947164 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5a28e0-197e-430e-996e-38f50ab0b723-maas-api-xks-ca\") pod \"maas-api-revoke-sub-130ee7b3-tjxzk\" (UID: \"9d5a28e0-197e-430e-996e-38f50ab0b723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk" Oct 30 18:13:25.963676 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:25.963639 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kpg25\" (UniqueName: \"kubernetes.io/projected/9d5a28e0-197e-430e-996e-38f50ab0b723-kube-api-access-kpg25\") pod \"maas-api-revoke-sub-130ee7b3-tjxzk\" (UID: \"9d5a28e0-197e-430e-996e-38f50ab0b723\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk" Oct 30 18:13:26.012806 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:26.012763 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk" Oct 30 18:13:26.311474 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:26.311443 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk"] Oct 30 18:13:26.314720 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:13:26.314680 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d5a28e0_197e_430e_996e_38f50ab0b723.slice/crio-ca76bd690e4ed9789b7ecec89c90290d9497397a3dcab659a83a8bb13901ea03 WatchSource:0}: Error finding container ca76bd690e4ed9789b7ecec89c90290d9497397a3dcab659a83a8bb13901ea03: Status 404 returned error can't find the container with id ca76bd690e4ed9789b7ecec89c90290d9497397a3dcab659a83a8bb13901ea03 Oct 30 18:13:26.379248 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:26.379207 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk"] Oct 30 18:13:26.384974 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:26.384948 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk" Oct 30 18:13:26.400374 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:26.400340 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk"] Oct 30 18:13:26.433939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:26.433902 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk" event={"ID":"9d5a28e0-197e-430e-996e-38f50ab0b723","Type":"ContainerStarted","Data":"ca76bd690e4ed9789b7ecec89c90290d9497397a3dcab659a83a8bb13901ea03"} Oct 30 18:13:26.550789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:26.550749 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf89l\" (UniqueName: \"kubernetes.io/projected/77678ec4-a4ae-4303-9961-ff06912bf20d-kube-api-access-rf89l\") pod \"maas-api-revoke-sub-36b1c2b3-snpjk\" (UID: \"77678ec4-a4ae-4303-9961-ff06912bf20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk" Oct 30 18:13:26.550956 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:26.550808 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77678ec4-a4ae-4303-9961-ff06912bf20d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36b1c2b3-snpjk\" (UID: \"77678ec4-a4ae-4303-9961-ff06912bf20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk" Oct 30 18:13:26.550956 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:26.550938 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77678ec4-a4ae-4303-9961-ff06912bf20d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36b1c2b3-snpjk\" (UID: \"77678ec4-a4ae-4303-9961-ff06912bf20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk" Oct 30 18:13:26.652445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:26.652413 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77678ec4-a4ae-4303-9961-ff06912bf20d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36b1c2b3-snpjk\" (UID: \"77678ec4-a4ae-4303-9961-ff06912bf20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk" Oct 30 18:13:26.652634 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:26.652495 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77678ec4-a4ae-4303-9961-ff06912bf20d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36b1c2b3-snpjk\" (UID: \"77678ec4-a4ae-4303-9961-ff06912bf20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk" Oct 30 18:13:26.652690 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:26.652651 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf89l\" (UniqueName: \"kubernetes.io/projected/77678ec4-a4ae-4303-9961-ff06912bf20d-kube-api-access-rf89l\") pod \"maas-api-revoke-sub-36b1c2b3-snpjk\" (UID: \"77678ec4-a4ae-4303-9961-ff06912bf20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk" Oct 30 18:13:26.653169 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:26.653144 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77678ec4-a4ae-4303-9961-ff06912bf20d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36b1c2b3-snpjk\" (UID: \"77678ec4-a4ae-4303-9961-ff06912bf20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk" Oct 30 18:13:26.654928 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:26.654908 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77678ec4-a4ae-4303-9961-ff06912bf20d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36b1c2b3-snpjk\" (UID: \"77678ec4-a4ae-4303-9961-ff06912bf20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk" Oct 30 18:13:26.671590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:26.671559 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf89l\" (UniqueName: \"kubernetes.io/projected/77678ec4-a4ae-4303-9961-ff06912bf20d-kube-api-access-rf89l\") pod \"maas-api-revoke-sub-36b1c2b3-snpjk\" (UID: \"77678ec4-a4ae-4303-9961-ff06912bf20d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk" Oct 30 18:13:26.701643 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:26.701608 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk" Oct 30 18:13:27.012632 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:27.012595 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk"] Oct 30 18:13:27.069427 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:13:27.069394 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:13:27.440962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:27.440928 5232 generic.go:356] "Generic (PLEG): container finished" podID="9d5a28e0-197e-430e-996e-38f50ab0b723" containerID="ff7f4cabfe259ee6b4131e4b515e509eb4a0aad1c74298af92fc504f4fe8d915" exitCode=0 Oct 30 18:13:27.441117 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:27.441020 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk" event={"ID":"9d5a28e0-197e-430e-996e-38f50ab0b723","Type":"ContainerDied","Data":"ff7f4cabfe259ee6b4131e4b515e509eb4a0aad1c74298af92fc504f4fe8d915"} Oct 30 18:13:27.442663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:27.442639 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk" event={"ID":"77678ec4-a4ae-4303-9961-ff06912bf20d","Type":"ContainerStarted","Data":"da5858690e7aa92f06d81b6f973e5cdd3c19532b948c38370221afc607425432"} Oct 30 18:13:27.442796 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:27.442678 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk" event={"ID":"77678ec4-a4ae-4303-9961-ff06912bf20d","Type":"ContainerStarted","Data":"e81c5682676e5628405f2b5eaa74680cbd208721c5405f42cd68103421410099"} Oct 30 18:13:27.517864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:27.517801 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk" podStartSLOduration=1.517783879 podStartE2EDuration="1.517783879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:27.516061418 +0000 UTC m=+706.832481739" watchObservedRunningTime="2026-10-30 18:13:27.517783879 +0000 UTC m=+706.834204177" Oct 30 18:13:28.450309 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.450277 5232 generic.go:356] "Generic (PLEG): container finished" podID="77678ec4-a4ae-4303-9961-ff06912bf20d" containerID="da5858690e7aa92f06d81b6f973e5cdd3c19532b948c38370221afc607425432" exitCode=0 Oct 30 18:13:28.450830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.450313 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk" event={"ID":"77678ec4-a4ae-4303-9961-ff06912bf20d","Type":"ContainerDied","Data":"da5858690e7aa92f06d81b6f973e5cdd3c19532b948c38370221afc607425432"} Oct 30 18:13:28.680685 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.680655 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk" Oct 30 18:13:28.778809 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.778772 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9d5a28e0-197e-430e-996e-38f50ab0b723-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5a28e0-197e-430e-996e-38f50ab0b723-maas-api-xks-ca\") pod \"9d5a28e0-197e-430e-996e-38f50ab0b723\" (UID: \"9d5a28e0-197e-430e-996e-38f50ab0b723\") " Oct 30 18:13:28.779009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.778865 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9d5a28e0-197e-430e-996e-38f50ab0b723-kube-api-access-kpg25\" (UniqueName: \"kubernetes.io/projected/9d5a28e0-197e-430e-996e-38f50ab0b723-kube-api-access-kpg25\") pod \"9d5a28e0-197e-430e-996e-38f50ab0b723\" (UID: \"9d5a28e0-197e-430e-996e-38f50ab0b723\") " Oct 30 18:13:28.779009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.778932 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9d5a28e0-197e-430e-996e-38f50ab0b723-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5a28e0-197e-430e-996e-38f50ab0b723-maas-api-openshift-ca\") pod \"9d5a28e0-197e-430e-996e-38f50ab0b723\" (UID: \"9d5a28e0-197e-430e-996e-38f50ab0b723\") " Oct 30 18:13:28.779311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.779284 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9d5a28e0-197e-430e-996e-38f50ab0b723-maas-api-openshift-ca" pod "9d5a28e0-197e-430e-996e-38f50ab0b723" (UID: "9d5a28e0-197e-430e-996e-38f50ab0b723"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:28.779451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.779375 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9d5a28e0-197e-430e-996e-38f50ab0b723-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.781048 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.781024 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9d5a28e0-197e-430e-996e-38f50ab0b723-maas-api-xks-ca" pod "9d5a28e0-197e-430e-996e-38f50ab0b723" (UID: "9d5a28e0-197e-430e-996e-38f50ab0b723"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:28.781129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.781051 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9d5a28e0-197e-430e-996e-38f50ab0b723-kube-api-access-kpg25" pod "9d5a28e0-197e-430e-996e-38f50ab0b723" (UID: "9d5a28e0-197e-430e-996e-38f50ab0b723"). InnerVolumeSpecName "kube-api-access-kpg25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:28.880325 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.880288 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9d5a28e0-197e-430e-996e-38f50ab0b723-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.880325 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.880319 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kpg25\" (UniqueName: \"kubernetes.io/projected/9d5a28e0-197e-430e-996e-38f50ab0b723-kube-api-access-kpg25\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:28.946613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.946583 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks"] Oct 30 18:13:28.947293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.947278 5232 cpu_manager.go:414] "containerMap: removing container" podUID="9d5a28e0-197e-430e-996e-38f50ab0b723" containerName="revoke-keys" Oct 30 18:13:28.947357 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.947295 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9d5a28e0-197e-430e-996e-38f50ab0b723" containerName="revoke-keys" Oct 30 18:13:28.947405 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.947396 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="9d5a28e0-197e-430e-996e-38f50ab0b723" containerName="revoke-keys" Oct 30 18:13:28.950824 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.950807 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks" Oct 30 18:13:28.967771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.967746 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks"] Oct 30 18:13:28.981190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.981164 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabcd183-716c-4012-8a6f-054b1cf65d47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eb098bf-ff6ks\" (UID: \"cabcd183-716c-4012-8a6f-054b1cf65d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks" Oct 30 18:13:28.981413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.981258 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabcd183-716c-4012-8a6f-054b1cf65d47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eb098bf-ff6ks\" (UID: \"cabcd183-716c-4012-8a6f-054b1cf65d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks" Oct 30 18:13:28.981413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:28.981376 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-knrfv\" (UniqueName: \"kubernetes.io/projected/cabcd183-716c-4012-8a6f-054b1cf65d47-kube-api-access-knrfv\") pod \"maas-api-revoke-sub-4eb098bf-ff6ks\" (UID: \"cabcd183-716c-4012-8a6f-054b1cf65d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks" Oct 30 18:13:29.082773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.082688 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabcd183-716c-4012-8a6f-054b1cf65d47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eb098bf-ff6ks\" (UID: \"cabcd183-716c-4012-8a6f-054b1cf65d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks" Oct 30 18:13:29.082922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.082803 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-knrfv\" (UniqueName: \"kubernetes.io/projected/cabcd183-716c-4012-8a6f-054b1cf65d47-kube-api-access-knrfv\") pod \"maas-api-revoke-sub-4eb098bf-ff6ks\" (UID: \"cabcd183-716c-4012-8a6f-054b1cf65d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks" Oct 30 18:13:29.082922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.082896 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabcd183-716c-4012-8a6f-054b1cf65d47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eb098bf-ff6ks\" (UID: \"cabcd183-716c-4012-8a6f-054b1cf65d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks" Oct 30 18:13:29.083422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.083404 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabcd183-716c-4012-8a6f-054b1cf65d47-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4eb098bf-ff6ks\" (UID: \"cabcd183-716c-4012-8a6f-054b1cf65d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks" Oct 30 18:13:29.085425 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.085403 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabcd183-716c-4012-8a6f-054b1cf65d47-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4eb098bf-ff6ks\" (UID: \"cabcd183-716c-4012-8a6f-054b1cf65d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks" Oct 30 18:13:29.101444 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.101417 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-knrfv\" (UniqueName: \"kubernetes.io/projected/cabcd183-716c-4012-8a6f-054b1cf65d47-kube-api-access-knrfv\") pod \"maas-api-revoke-sub-4eb098bf-ff6ks\" (UID: \"cabcd183-716c-4012-8a6f-054b1cf65d47\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks" Oct 30 18:13:29.266369 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.266332 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks" Oct 30 18:13:29.458227 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.458132 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk" Oct 30 18:13:29.458227 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.458172 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk" event={"ID":"9d5a28e0-197e-430e-996e-38f50ab0b723","Type":"ContainerDied","Data":"ca76bd690e4ed9789b7ecec89c90290d9497397a3dcab659a83a8bb13901ea03"} Oct 30 18:13:29.458227 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.458225 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca76bd690e4ed9789b7ecec89c90290d9497397a3dcab659a83a8bb13901ea03" Oct 30 18:13:29.564647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.564555 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks"] Oct 30 18:13:29.566971 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:13:29.566938 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcabcd183_716c_4012_8a6f_054b1cf65d47.slice/crio-33ba93d1fa06c9a093893efcd464dda5aef97837d4bd90f17ee424ebf73b3e57 WatchSource:0}: Error finding container 33ba93d1fa06c9a093893efcd464dda5aef97837d4bd90f17ee424ebf73b3e57: Status 404 returned error can't find the container with id 33ba93d1fa06c9a093893efcd464dda5aef97837d4bd90f17ee424ebf73b3e57 Oct 30 18:13:29.587907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.587876 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk" Oct 30 18:13:29.689615 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.689581 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/77678ec4-a4ae-4303-9961-ff06912bf20d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77678ec4-a4ae-4303-9961-ff06912bf20d-maas-api-openshift-ca\") pod \"77678ec4-a4ae-4303-9961-ff06912bf20d\" (UID: \"77678ec4-a4ae-4303-9961-ff06912bf20d\") " Oct 30 18:13:29.689769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.689635 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77678ec4-a4ae-4303-9961-ff06912bf20d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77678ec4-a4ae-4303-9961-ff06912bf20d-maas-api-xks-ca\") pod \"77678ec4-a4ae-4303-9961-ff06912bf20d\" (UID: \"77678ec4-a4ae-4303-9961-ff06912bf20d\") " Oct 30 18:13:29.689769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.689742 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77678ec4-a4ae-4303-9961-ff06912bf20d-kube-api-access-rf89l\" (UniqueName: \"kubernetes.io/projected/77678ec4-a4ae-4303-9961-ff06912bf20d-kube-api-access-rf89l\") pod \"77678ec4-a4ae-4303-9961-ff06912bf20d\" (UID: \"77678ec4-a4ae-4303-9961-ff06912bf20d\") " Oct 30 18:13:29.689952 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.689929 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/77678ec4-a4ae-4303-9961-ff06912bf20d-maas-api-openshift-ca" pod "77678ec4-a4ae-4303-9961-ff06912bf20d" (UID: "77678ec4-a4ae-4303-9961-ff06912bf20d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:29.690183 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.690168 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/77678ec4-a4ae-4303-9961-ff06912bf20d-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:29.691831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.691803 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77678ec4-a4ae-4303-9961-ff06912bf20d-maas-api-xks-ca" pod "77678ec4-a4ae-4303-9961-ff06912bf20d" (UID: "77678ec4-a4ae-4303-9961-ff06912bf20d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:29.691933 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.691899 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77678ec4-a4ae-4303-9961-ff06912bf20d-kube-api-access-rf89l" pod "77678ec4-a4ae-4303-9961-ff06912bf20d" (UID: "77678ec4-a4ae-4303-9961-ff06912bf20d"). InnerVolumeSpecName "kube-api-access-rf89l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:29.791009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.790977 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf89l\" (UniqueName: \"kubernetes.io/projected/77678ec4-a4ae-4303-9961-ff06912bf20d-kube-api-access-rf89l\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:29.791009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:29.791007 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/77678ec4-a4ae-4303-9961-ff06912bf20d-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:30.464871 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:30.464847 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk" Oct 30 18:13:30.465296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:30.464882 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk" event={"ID":"77678ec4-a4ae-4303-9961-ff06912bf20d","Type":"ContainerDied","Data":"e81c5682676e5628405f2b5eaa74680cbd208721c5405f42cd68103421410099"} Oct 30 18:13:30.465296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:30.464935 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e81c5682676e5628405f2b5eaa74680cbd208721c5405f42cd68103421410099" Oct 30 18:13:30.466590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:30.466566 5232 generic.go:356] "Generic (PLEG): container finished" podID="cabcd183-716c-4012-8a6f-054b1cf65d47" containerID="34a2903fed7c330ca21c07983552726aa0b1d00fea647b0223459d789909d6ec" exitCode=0 Oct 30 18:13:30.466697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:30.466638 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks" event={"ID":"cabcd183-716c-4012-8a6f-054b1cf65d47","Type":"ContainerDied","Data":"34a2903fed7c330ca21c07983552726aa0b1d00fea647b0223459d789909d6ec"} Oct 30 18:13:30.466697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:30.466689 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks" event={"ID":"cabcd183-716c-4012-8a6f-054b1cf65d47","Type":"ContainerStarted","Data":"33ba93d1fa06c9a093893efcd464dda5aef97837d4bd90f17ee424ebf73b3e57"} Oct 30 18:13:30.972195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:30.972164 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t"] Oct 30 18:13:30.972761 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:30.972744 5232 cpu_manager.go:414] "containerMap: removing container" podUID="77678ec4-a4ae-4303-9961-ff06912bf20d" containerName="revoke-keys" Oct 30 18:13:30.972761 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:30.972762 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77678ec4-a4ae-4303-9961-ff06912bf20d" containerName="revoke-keys" Oct 30 18:13:30.972955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:30.972878 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="77678ec4-a4ae-4303-9961-ff06912bf20d" containerName="revoke-keys" Oct 30 18:13:30.976126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:30.976106 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t" Oct 30 18:13:30.995310 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:30.995287 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t"] Oct 30 18:13:31.009690 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.009664 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd235ee3-pvj2t\" (UID: \"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t" Oct 30 18:13:31.009841 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.009752 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd235ee3-pvj2t\" (UID: \"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t" Oct 30 18:13:31.009930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.009910 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pbpv\" (UniqueName: \"kubernetes.io/projected/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-kube-api-access-9pbpv\") pod \"maas-api-revoke-sub-cd235ee3-pvj2t\" (UID: \"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t" Oct 30 18:13:31.095208 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.095168 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl"] Oct 30 18:13:31.099033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.099007 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl" Oct 30 18:13:31.111456 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.111431 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pbpv\" (UniqueName: \"kubernetes.io/projected/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-kube-api-access-9pbpv\") pod \"maas-api-revoke-sub-cd235ee3-pvj2t\" (UID: \"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t" Oct 30 18:13:31.111603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.111477 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056a0fd0-edc7-4b1a-a06a-8f794146edd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0a2786d-hmrsl\" (UID: \"056a0fd0-edc7-4b1a-a06a-8f794146edd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl" Oct 30 18:13:31.111683 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.111650 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd235ee3-pvj2t\" (UID: \"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t" Oct 30 18:13:31.111761 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.111736 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbmq8\" (UniqueName: \"kubernetes.io/projected/056a0fd0-edc7-4b1a-a06a-8f794146edd0-kube-api-access-gbmq8\") pod \"maas-api-revoke-sub-c0a2786d-hmrsl\" (UID: \"056a0fd0-edc7-4b1a-a06a-8f794146edd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl" Oct 30 18:13:31.111869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.111845 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd235ee3-pvj2t\" (UID: \"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t" Oct 30 18:13:31.111922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.111887 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056a0fd0-edc7-4b1a-a06a-8f794146edd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0a2786d-hmrsl\" (UID: \"056a0fd0-edc7-4b1a-a06a-8f794146edd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl" Oct 30 18:13:31.112251 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.112218 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd235ee3-pvj2t\" (UID: \"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t" Oct 30 18:13:31.114842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.114765 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd235ee3-pvj2t\" (UID: \"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t" Oct 30 18:13:31.117559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.117532 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl"] Oct 30 18:13:31.130352 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.130321 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pbpv\" (UniqueName: \"kubernetes.io/projected/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-kube-api-access-9pbpv\") pod \"maas-api-revoke-sub-cd235ee3-pvj2t\" (UID: \"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t" Oct 30 18:13:31.213566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.213500 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbmq8\" (UniqueName: \"kubernetes.io/projected/056a0fd0-edc7-4b1a-a06a-8f794146edd0-kube-api-access-gbmq8\") pod \"maas-api-revoke-sub-c0a2786d-hmrsl\" (UID: \"056a0fd0-edc7-4b1a-a06a-8f794146edd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl" Oct 30 18:13:31.213730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.213593 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056a0fd0-edc7-4b1a-a06a-8f794146edd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0a2786d-hmrsl\" (UID: \"056a0fd0-edc7-4b1a-a06a-8f794146edd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl" Oct 30 18:13:31.213730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.213692 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056a0fd0-edc7-4b1a-a06a-8f794146edd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0a2786d-hmrsl\" (UID: \"056a0fd0-edc7-4b1a-a06a-8f794146edd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl" Oct 30 18:13:31.214282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.214251 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056a0fd0-edc7-4b1a-a06a-8f794146edd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c0a2786d-hmrsl\" (UID: \"056a0fd0-edc7-4b1a-a06a-8f794146edd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl" Oct 30 18:13:31.216209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.216187 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056a0fd0-edc7-4b1a-a06a-8f794146edd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c0a2786d-hmrsl\" (UID: \"056a0fd0-edc7-4b1a-a06a-8f794146edd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl" Oct 30 18:13:31.232777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.232715 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbmq8\" (UniqueName: \"kubernetes.io/projected/056a0fd0-edc7-4b1a-a06a-8f794146edd0-kube-api-access-gbmq8\") pod \"maas-api-revoke-sub-c0a2786d-hmrsl\" (UID: \"056a0fd0-edc7-4b1a-a06a-8f794146edd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl" Oct 30 18:13:31.289496 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.289463 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t" Oct 30 18:13:31.413108 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.413079 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl" Oct 30 18:13:31.598886 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.598851 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t"] Oct 30 18:13:31.650807 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.650785 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks" Oct 30 18:13:31.713807 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.712850 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl"] Oct 30 18:13:31.716876 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:13:31.716847 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod056a0fd0_edc7_4b1a_a06a_8f794146edd0.slice/crio-4284c208ddd364b61e1ccfa01bf522d285c2c746b18edb0b237d192c43f7ffe1 WatchSource:0}: Error finding container 4284c208ddd364b61e1ccfa01bf522d285c2c746b18edb0b237d192c43f7ffe1: Status 404 returned error can't find the container with id 4284c208ddd364b61e1ccfa01bf522d285c2c746b18edb0b237d192c43f7ffe1 Oct 30 18:13:31.723809 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.723780 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cabcd183-716c-4012-8a6f-054b1cf65d47-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabcd183-716c-4012-8a6f-054b1cf65d47-maas-api-xks-ca\") pod \"cabcd183-716c-4012-8a6f-054b1cf65d47\" (UID: \"cabcd183-716c-4012-8a6f-054b1cf65d47\") " Oct 30 18:13:31.723927 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.723909 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cabcd183-716c-4012-8a6f-054b1cf65d47-kube-api-access-knrfv\" (UniqueName: \"kubernetes.io/projected/cabcd183-716c-4012-8a6f-054b1cf65d47-kube-api-access-knrfv\") pod \"cabcd183-716c-4012-8a6f-054b1cf65d47\" (UID: \"cabcd183-716c-4012-8a6f-054b1cf65d47\") " Oct 30 18:13:31.724031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.724015 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cabcd183-716c-4012-8a6f-054b1cf65d47-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabcd183-716c-4012-8a6f-054b1cf65d47-maas-api-openshift-ca\") pod \"cabcd183-716c-4012-8a6f-054b1cf65d47\" (UID: \"cabcd183-716c-4012-8a6f-054b1cf65d47\") " Oct 30 18:13:31.724442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.724401 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cabcd183-716c-4012-8a6f-054b1cf65d47-maas-api-openshift-ca" pod "cabcd183-716c-4012-8a6f-054b1cf65d47" (UID: "cabcd183-716c-4012-8a6f-054b1cf65d47"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:31.726119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.726091 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cabcd183-716c-4012-8a6f-054b1cf65d47-kube-api-access-knrfv" pod "cabcd183-716c-4012-8a6f-054b1cf65d47" (UID: "cabcd183-716c-4012-8a6f-054b1cf65d47"). InnerVolumeSpecName "kube-api-access-knrfv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:31.726378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.726357 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cabcd183-716c-4012-8a6f-054b1cf65d47-maas-api-xks-ca" pod "cabcd183-716c-4012-8a6f-054b1cf65d47" (UID: "cabcd183-716c-4012-8a6f-054b1cf65d47"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:31.826036 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.826004 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cabcd183-716c-4012-8a6f-054b1cf65d47-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:31.826036 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.826038 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cabcd183-716c-4012-8a6f-054b1cf65d47-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:31.826206 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:31.826052 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-knrfv\" (UniqueName: \"kubernetes.io/projected/cabcd183-716c-4012-8a6f-054b1cf65d47-kube-api-access-knrfv\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:32.489764 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:32.489722 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t" event={"ID":"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4","Type":"ContainerStarted","Data":"c44a0a10e7651b3d5c3534aaf56f82c97e9413521dc9874420f5f78ed2486903"} Oct 30 18:13:32.489937 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:32.489790 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t" event={"ID":"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4","Type":"ContainerStarted","Data":"12218f3b1909f5cbf0d8591ad52d352e9ae166d6bc94626486ce328b5ff09f5d"} Oct 30 18:13:32.491414 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:32.491389 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl" event={"ID":"056a0fd0-edc7-4b1a-a06a-8f794146edd0","Type":"ContainerStarted","Data":"1b0828f41bdad4df2e06bfc24fe7f9ab37044d52a7b52fd49f969f68a1b82b39"} Oct 30 18:13:32.491557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:32.491429 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl" event={"ID":"056a0fd0-edc7-4b1a-a06a-8f794146edd0","Type":"ContainerStarted","Data":"4284c208ddd364b61e1ccfa01bf522d285c2c746b18edb0b237d192c43f7ffe1"} Oct 30 18:13:32.492633 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:32.492614 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks" Oct 30 18:13:32.492742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:32.492664 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks" event={"ID":"cabcd183-716c-4012-8a6f-054b1cf65d47","Type":"ContainerDied","Data":"33ba93d1fa06c9a093893efcd464dda5aef97837d4bd90f17ee424ebf73b3e57"} Oct 30 18:13:32.492742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:32.492709 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="33ba93d1fa06c9a093893efcd464dda5aef97837d4bd90f17ee424ebf73b3e57" Oct 30 18:13:32.528491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:32.528440 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t" podStartSLOduration=2.528427227 podStartE2EDuration="2.528427227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:32.526974014 +0000 UTC m=+711.843394335" watchObservedRunningTime="2026-10-30 18:13:32.528427227 +0000 UTC m=+711.844847548" Oct 30 18:13:32.563830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:32.563778 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl" podStartSLOduration=1.5637463889999998 podStartE2EDuration="1.563746389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:32.563043859 +0000 UTC m=+711.879464181" watchObservedRunningTime="2026-10-30 18:13:32.563746389 +0000 UTC m=+711.880166711" Oct 30 18:13:33.512884 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:33.512840 5232 generic.go:356] "Generic (PLEG): container finished" podID="9fb55b43-dcfe-4a5d-afba-f720ec95fbf4" containerID="c44a0a10e7651b3d5c3534aaf56f82c97e9413521dc9874420f5f78ed2486903" exitCode=0 Oct 30 18:13:33.513368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:33.512964 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t" event={"ID":"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4","Type":"ContainerDied","Data":"c44a0a10e7651b3d5c3534aaf56f82c97e9413521dc9874420f5f78ed2486903"} Oct 30 18:13:33.527835 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:33.527757 5232 generic.go:356] "Generic (PLEG): container finished" podID="056a0fd0-edc7-4b1a-a06a-8f794146edd0" containerID="1b0828f41bdad4df2e06bfc24fe7f9ab37044d52a7b52fd49f969f68a1b82b39" exitCode=0 Oct 30 18:13:33.527835 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:33.527824 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl" event={"ID":"056a0fd0-edc7-4b1a-a06a-8f794146edd0","Type":"ContainerDied","Data":"1b0828f41bdad4df2e06bfc24fe7f9ab37044d52a7b52fd49f969f68a1b82b39"} Oct 30 18:13:34.344479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.344432 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg"] Oct 30 18:13:34.345410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.345386 5232 cpu_manager.go:414] "containerMap: removing container" podUID="cabcd183-716c-4012-8a6f-054b1cf65d47" containerName="revoke-keys" Oct 30 18:13:34.345410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.345411 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cabcd183-716c-4012-8a6f-054b1cf65d47" containerName="revoke-keys" Oct 30 18:13:34.345743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.345718 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="cabcd183-716c-4012-8a6f-054b1cf65d47" containerName="revoke-keys" Oct 30 18:13:34.351597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.351570 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg" Oct 30 18:13:34.365546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.365499 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg"] Oct 30 18:13:34.453654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.453619 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc03de8a-fe28-4db9-9f5b-730beac01d95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20f778b3-dnxfg\" (UID: \"fc03de8a-fe28-4db9-9f5b-730beac01d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg" Oct 30 18:13:34.453817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.453742 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc03de8a-fe28-4db9-9f5b-730beac01d95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20f778b3-dnxfg\" (UID: \"fc03de8a-fe28-4db9-9f5b-730beac01d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg" Oct 30 18:13:34.453860 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.453850 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4qqx\" (UniqueName: \"kubernetes.io/projected/fc03de8a-fe28-4db9-9f5b-730beac01d95-kube-api-access-z4qqx\") pod \"maas-api-revoke-sub-20f778b3-dnxfg\" (UID: \"fc03de8a-fe28-4db9-9f5b-730beac01d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg" Oct 30 18:13:34.555242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.555207 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4qqx\" (UniqueName: \"kubernetes.io/projected/fc03de8a-fe28-4db9-9f5b-730beac01d95-kube-api-access-z4qqx\") pod \"maas-api-revoke-sub-20f778b3-dnxfg\" (UID: \"fc03de8a-fe28-4db9-9f5b-730beac01d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg" Oct 30 18:13:34.555728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.555302 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc03de8a-fe28-4db9-9f5b-730beac01d95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20f778b3-dnxfg\" (UID: \"fc03de8a-fe28-4db9-9f5b-730beac01d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg" Oct 30 18:13:34.555728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.555404 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc03de8a-fe28-4db9-9f5b-730beac01d95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20f778b3-dnxfg\" (UID: \"fc03de8a-fe28-4db9-9f5b-730beac01d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg" Oct 30 18:13:34.556081 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.556052 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc03de8a-fe28-4db9-9f5b-730beac01d95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-20f778b3-dnxfg\" (UID: \"fc03de8a-fe28-4db9-9f5b-730beac01d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg" Oct 30 18:13:34.557992 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.557971 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc03de8a-fe28-4db9-9f5b-730beac01d95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-20f778b3-dnxfg\" (UID: \"fc03de8a-fe28-4db9-9f5b-730beac01d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg" Oct 30 18:13:34.574730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.574701 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4qqx\" (UniqueName: \"kubernetes.io/projected/fc03de8a-fe28-4db9-9f5b-730beac01d95-kube-api-access-z4qqx\") pod \"maas-api-revoke-sub-20f778b3-dnxfg\" (UID: \"fc03de8a-fe28-4db9-9f5b-730beac01d95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg" Oct 30 18:13:34.667688 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.667653 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg" Oct 30 18:13:34.718266 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.718238 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t" Oct 30 18:13:34.730074 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.729992 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl" Oct 30 18:13:34.767064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.763694 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/056a0fd0-edc7-4b1a-a06a-8f794146edd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056a0fd0-edc7-4b1a-a06a-8f794146edd0-maas-api-xks-ca\") pod \"056a0fd0-edc7-4b1a-a06a-8f794146edd0\" (UID: \"056a0fd0-edc7-4b1a-a06a-8f794146edd0\") " Oct 30 18:13:34.767064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.763839 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-kube-api-access-9pbpv\" (UniqueName: \"kubernetes.io/projected/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-kube-api-access-9pbpv\") pod \"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4\" (UID: \"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4\") " Oct 30 18:13:34.767064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.763906 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-maas-api-openshift-ca\") pod \"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4\" (UID: \"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4\") " Oct 30 18:13:34.767064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.763929 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/056a0fd0-edc7-4b1a-a06a-8f794146edd0-kube-api-access-gbmq8\" (UniqueName: \"kubernetes.io/projected/056a0fd0-edc7-4b1a-a06a-8f794146edd0-kube-api-access-gbmq8\") pod \"056a0fd0-edc7-4b1a-a06a-8f794146edd0\" (UID: \"056a0fd0-edc7-4b1a-a06a-8f794146edd0\") " Oct 30 18:13:34.767064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.763959 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/056a0fd0-edc7-4b1a-a06a-8f794146edd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056a0fd0-edc7-4b1a-a06a-8f794146edd0-maas-api-openshift-ca\") pod \"056a0fd0-edc7-4b1a-a06a-8f794146edd0\" (UID: \"056a0fd0-edc7-4b1a-a06a-8f794146edd0\") " Oct 30 18:13:34.767064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.763979 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-maas-api-xks-ca\") pod \"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4\" (UID: \"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4\") " Oct 30 18:13:34.767064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.764904 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-maas-api-openshift-ca" pod "9fb55b43-dcfe-4a5d-afba-f720ec95fbf4" (UID: "9fb55b43-dcfe-4a5d-afba-f720ec95fbf4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:34.767392 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.767190 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/056a0fd0-edc7-4b1a-a06a-8f794146edd0-maas-api-openshift-ca" pod "056a0fd0-edc7-4b1a-a06a-8f794146edd0" (UID: "056a0fd0-edc7-4b1a-a06a-8f794146edd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:34.769730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.769570 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/056a0fd0-edc7-4b1a-a06a-8f794146edd0-kube-api-access-gbmq8" pod "056a0fd0-edc7-4b1a-a06a-8f794146edd0" (UID: "056a0fd0-edc7-4b1a-a06a-8f794146edd0"). InnerVolumeSpecName "kube-api-access-gbmq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:34.769730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.769603 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/056a0fd0-edc7-4b1a-a06a-8f794146edd0-maas-api-xks-ca" pod "056a0fd0-edc7-4b1a-a06a-8f794146edd0" (UID: "056a0fd0-edc7-4b1a-a06a-8f794146edd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:34.769730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.769682 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-maas-api-xks-ca" pod "9fb55b43-dcfe-4a5d-afba-f720ec95fbf4" (UID: "9fb55b43-dcfe-4a5d-afba-f720ec95fbf4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:34.770931 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.770907 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-kube-api-access-9pbpv" pod "9fb55b43-dcfe-4a5d-afba-f720ec95fbf4" (UID: "9fb55b43-dcfe-4a5d-afba-f720ec95fbf4"). InnerVolumeSpecName "kube-api-access-9pbpv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:34.865833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.865741 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:34.865833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.865771 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbmq8\" (UniqueName: \"kubernetes.io/projected/056a0fd0-edc7-4b1a-a06a-8f794146edd0-kube-api-access-gbmq8\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:34.865833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.865786 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/056a0fd0-edc7-4b1a-a06a-8f794146edd0-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:34.865833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.865799 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:34.865833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.865813 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/056a0fd0-edc7-4b1a-a06a-8f794146edd0-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:34.865833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.865824 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pbpv\" (UniqueName: \"kubernetes.io/projected/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4-kube-api-access-9pbpv\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:34.979000 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:34.978892 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg"] Oct 30 18:13:34.982679 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:13:34.982648 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc03de8a_fe28_4db9_9f5b_730beac01d95.slice/crio-646806fbe3497784433d6724f61bf9fd0c187f8557ac8f98668e90cd2f070fda WatchSource:0}: Error finding container 646806fbe3497784433d6724f61bf9fd0c187f8557ac8f98668e90cd2f070fda: Status 404 returned error can't find the container with id 646806fbe3497784433d6724f61bf9fd0c187f8557ac8f98668e90cd2f070fda Oct 30 18:13:35.595673 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:35.594698 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t" event={"ID":"9fb55b43-dcfe-4a5d-afba-f720ec95fbf4","Type":"ContainerDied","Data":"12218f3b1909f5cbf0d8591ad52d352e9ae166d6bc94626486ce328b5ff09f5d"} Oct 30 18:13:35.595673 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:35.594767 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="12218f3b1909f5cbf0d8591ad52d352e9ae166d6bc94626486ce328b5ff09f5d" Oct 30 18:13:35.595673 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:35.594836 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t" Oct 30 18:13:35.602911 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:35.602875 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl" event={"ID":"056a0fd0-edc7-4b1a-a06a-8f794146edd0","Type":"ContainerDied","Data":"4284c208ddd364b61e1ccfa01bf522d285c2c746b18edb0b237d192c43f7ffe1"} Oct 30 18:13:35.619578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:35.603139 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4284c208ddd364b61e1ccfa01bf522d285c2c746b18edb0b237d192c43f7ffe1" Oct 30 18:13:35.619578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:35.603239 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl" Oct 30 18:13:35.640549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:35.640101 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg" event={"ID":"fc03de8a-fe28-4db9-9f5b-730beac01d95","Type":"ContainerStarted","Data":"a7bd44c1c0fa0746da65b5f1f874af07a9903f1c66266215343f71f867dcc6fa"} Oct 30 18:13:35.640549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:35.640170 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg" event={"ID":"fc03de8a-fe28-4db9-9f5b-730beac01d95","Type":"ContainerStarted","Data":"646806fbe3497784433d6724f61bf9fd0c187f8557ac8f98668e90cd2f070fda"} Oct 30 18:13:36.703536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:36.688763 5232 generic.go:356] "Generic (PLEG): container finished" podID="fc03de8a-fe28-4db9-9f5b-730beac01d95" containerID="a7bd44c1c0fa0746da65b5f1f874af07a9903f1c66266215343f71f867dcc6fa" exitCode=0 Oct 30 18:13:36.703536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:36.688812 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg" event={"ID":"fc03de8a-fe28-4db9-9f5b-730beac01d95","Type":"ContainerDied","Data":"a7bd44c1c0fa0746da65b5f1f874af07a9903f1c66266215343f71f867dcc6fa"} Oct 30 18:13:38.227101 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.226719 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg" Oct 30 18:13:38.397243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.397202 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc03de8a-fe28-4db9-9f5b-730beac01d95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc03de8a-fe28-4db9-9f5b-730beac01d95-maas-api-openshift-ca\") pod \"fc03de8a-fe28-4db9-9f5b-730beac01d95\" (UID: \"fc03de8a-fe28-4db9-9f5b-730beac01d95\") " Oct 30 18:13:38.397559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.397534 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc03de8a-fe28-4db9-9f5b-730beac01d95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc03de8a-fe28-4db9-9f5b-730beac01d95-maas-api-xks-ca\") pod \"fc03de8a-fe28-4db9-9f5b-730beac01d95\" (UID: \"fc03de8a-fe28-4db9-9f5b-730beac01d95\") " Oct 30 18:13:38.397860 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.397842 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc03de8a-fe28-4db9-9f5b-730beac01d95-kube-api-access-z4qqx\" (UniqueName: \"kubernetes.io/projected/fc03de8a-fe28-4db9-9f5b-730beac01d95-kube-api-access-z4qqx\") pod \"fc03de8a-fe28-4db9-9f5b-730beac01d95\" (UID: \"fc03de8a-fe28-4db9-9f5b-730beac01d95\") " Oct 30 18:13:38.399362 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.399335 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc03de8a-fe28-4db9-9f5b-730beac01d95-maas-api-openshift-ca" pod "fc03de8a-fe28-4db9-9f5b-730beac01d95" (UID: "fc03de8a-fe28-4db9-9f5b-730beac01d95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:38.403198 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.403167 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc03de8a-fe28-4db9-9f5b-730beac01d95-maas-api-xks-ca" pod "fc03de8a-fe28-4db9-9f5b-730beac01d95" (UID: "fc03de8a-fe28-4db9-9f5b-730beac01d95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:38.403761 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.403739 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc03de8a-fe28-4db9-9f5b-730beac01d95-kube-api-access-z4qqx" pod "fc03de8a-fe28-4db9-9f5b-730beac01d95" (UID: "fc03de8a-fe28-4db9-9f5b-730beac01d95"). InnerVolumeSpecName "kube-api-access-z4qqx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:38.505484 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.505450 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4qqx\" (UniqueName: \"kubernetes.io/projected/fc03de8a-fe28-4db9-9f5b-730beac01d95-kube-api-access-z4qqx\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:38.505721 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.505703 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc03de8a-fe28-4db9-9f5b-730beac01d95-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:38.505827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.505815 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc03de8a-fe28-4db9-9f5b-730beac01d95-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:38.796067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.795724 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2"] Oct 30 18:13:38.796542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.796493 5232 cpu_manager.go:414] "containerMap: removing container" podUID="056a0fd0-edc7-4b1a-a06a-8f794146edd0" containerName="revoke-keys" Oct 30 18:13:38.796669 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.796546 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="056a0fd0-edc7-4b1a-a06a-8f794146edd0" containerName="revoke-keys" Oct 30 18:13:38.796669 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.796565 5232 cpu_manager.go:414] "containerMap: removing container" podUID="9fb55b43-dcfe-4a5d-afba-f720ec95fbf4" containerName="revoke-keys" Oct 30 18:13:38.796669 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.796571 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9fb55b43-dcfe-4a5d-afba-f720ec95fbf4" containerName="revoke-keys" Oct 30 18:13:38.796669 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.796579 5232 cpu_manager.go:414] "containerMap: removing container" podUID="fc03de8a-fe28-4db9-9f5b-730beac01d95" containerName="revoke-keys" Oct 30 18:13:38.796669 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.796587 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc03de8a-fe28-4db9-9f5b-730beac01d95" containerName="revoke-keys" Oct 30 18:13:38.797700 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.796762 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="056a0fd0-edc7-4b1a-a06a-8f794146edd0" containerName="revoke-keys" Oct 30 18:13:38.797700 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.796776 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="9fb55b43-dcfe-4a5d-afba-f720ec95fbf4" containerName="revoke-keys" Oct 30 18:13:38.797700 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.796784 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc03de8a-fe28-4db9-9f5b-730beac01d95" containerName="revoke-keys" Oct 30 18:13:38.800224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.800202 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2"] Oct 30 18:13:38.800368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.800348 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.800922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.800903 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg" Oct 30 18:13:38.804919 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.804891 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg" event={"ID":"fc03de8a-fe28-4db9-9f5b-730beac01d95","Type":"ContainerDied","Data":"646806fbe3497784433d6724f61bf9fd0c187f8557ac8f98668e90cd2f070fda"} Oct 30 18:13:38.805026 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.804938 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="646806fbe3497784433d6724f61bf9fd0c187f8557ac8f98668e90cd2f070fda" Oct 30 18:13:38.805026 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.805009 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:38.809275 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.809251 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.809363 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.809295 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.809465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.809438 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.815534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.809569 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tg85b\" (UniqueName: \"kubernetes.io/projected/45299d3c-52f3-4d0c-8dd3-af63948cf587-kube-api-access-tg85b\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.815534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.809656 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.815534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.809697 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.815534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.809764 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.815534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.809795 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45299d3c-52f3-4d0c-8dd3-af63948cf587-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.815534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.809869 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.864174 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.864105 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2"] Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.910848 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.910909 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.910936 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.910968 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.911002 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.911067 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.911112 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c7a6701-721d-448e-af9a-10216921bf48-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.911151 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c7a6701-721d-448e-af9a-10216921bf48-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.911186 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tg85b\" (UniqueName: \"kubernetes.io/projected/45299d3c-52f3-4d0c-8dd3-af63948cf587-kube-api-access-tg85b\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.911209 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.911280 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.911311 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.911378 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c7a6701-721d-448e-af9a-10216921bf48-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.911404 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.911432 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrk6k\" (UniqueName: \"kubernetes.io/projected/6c7a6701-721d-448e-af9a-10216921bf48-kube-api-access-mrk6k\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.911463 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45299d3c-52f3-4d0c-8dd3-af63948cf587-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.911490 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.922188 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.922988 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.923230 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.923456 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.924390 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.927590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.925035 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45299d3c-52f3-4d0c-8dd3-af63948cf587-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:38.993551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:38.993458 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tg85b\" (UniqueName: \"kubernetes.io/projected/45299d3c-52f3-4d0c-8dd3-af63948cf587-kube-api-access-tg85b\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:39.004559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.003845 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2"] Oct 30 18:13:39.004769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.004745 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:39.007530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.007475 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:39.007918 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.007896 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:39.071839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.054228 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c7a6701-721d-448e-af9a-10216921bf48-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.071839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.054285 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrk6k\" (UniqueName: \"kubernetes.io/projected/6c7a6701-721d-448e-af9a-10216921bf48-kube-api-access-mrk6k\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.071839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.054322 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.071839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.054367 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.071839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.054425 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.071839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.054449 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.071839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.054572 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c7a6701-721d-448e-af9a-10216921bf48-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.071839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.054621 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c7a6701-721d-448e-af9a-10216921bf48-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.071839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.054656 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.071839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.055250 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.071839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.056056 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.071839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.056298 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.071839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.056547 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.077824 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.077644 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:39.097301 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.096668 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c7a6701-721d-448e-af9a-10216921bf48-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.101015 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.100942 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c7a6701-721d-448e-af9a-10216921bf48-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.137533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.135269 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrk6k\" (UniqueName: \"kubernetes.io/projected/6c7a6701-721d-448e-af9a-10216921bf48-kube-api-access-mrk6k\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.137533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.135558 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.153575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.145072 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c7a6701-721d-448e-af9a-10216921bf48-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.378459 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.378428 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:39.594301 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.594279 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2"] Oct 30 18:13:39.596407 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:13:39.596364 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod45299d3c_52f3_4d0c_8dd3_af63948cf587.slice/crio-5804ea074e682766625ec940d6831f6d90be07595591917865df1510c6efa876 WatchSource:0}: Error finding container 5804ea074e682766625ec940d6831f6d90be07595591917865df1510c6efa876: Status 404 returned error can't find the container with id 5804ea074e682766625ec940d6831f6d90be07595591917865df1510c6efa876 Oct 30 18:13:39.598937 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.598892 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:13:39.599043 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.598973 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:13:39.599043 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.599021 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:13:39.693069 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.693035 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2"] Oct 30 18:13:39.710782 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.710744 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:13:39.711275 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.711251 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:13:39.711428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.711411 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:13:39.816281 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.816248 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" event={"ID":"6c7a6701-721d-448e-af9a-10216921bf48","Type":"ContainerStarted","Data":"ebfe8f0f900a962bd67566879a141eb1a901c596afa87da0b72d4c79f97cb871"} Oct 30 18:13:39.818219 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.818074 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" event={"ID":"45299d3c-52f3-4d0c-8dd3-af63948cf587","Type":"ContainerStarted","Data":"66335af6ba8311ec95c88d451a1a62cfa8c560aa3d6585bcc70d99e49ca27869"} Oct 30 18:13:39.818219 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.818119 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" event={"ID":"45299d3c-52f3-4d0c-8dd3-af63948cf587","Type":"ContainerStarted","Data":"5804ea074e682766625ec940d6831f6d90be07595591917865df1510c6efa876"} Oct 30 18:13:39.862152 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:39.862059 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" podStartSLOduration=1.8620459089999999 podStartE2EDuration="1.862045909s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:39.859214289 +0000 UTC m=+719.175634622" watchObservedRunningTime="2026-10-30 18:13:39.862045909 +0000 UTC m=+719.178466229" Oct 30 18:13:40.095354 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:40.095321 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:40.096973 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:40.096947 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:13:40.097089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:40.096993 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" podUID="45299d3c-52f3-4d0c-8dd3-af63948cf587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 18:13:40.876467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:40.876416 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" event={"ID":"6c7a6701-721d-448e-af9a-10216921bf48","Type":"ContainerStarted","Data":"8eb2dc9ae29604190a5d4a178668d778f78580546111260f2e2e59af72d54162"} Oct 30 18:13:40.966533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:40.962946 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" podStartSLOduration=2.962921705 podStartE2EDuration="2.962921705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:40.930057349 +0000 UTC m=+720.246477671" watchObservedRunningTime="2026-10-30 18:13:40.962921705 +0000 UTC m=+720.279342029" Oct 30 18:13:41.096233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:41.096205 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:13:41.096465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:41.096433 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" podUID="45299d3c-52f3-4d0c-8dd3-af63948cf587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 18:13:41.379478 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:41.379293 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:41.381049 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:41.381019 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:13:41.381181 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:41.381066 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" podUID="6c7a6701-721d-448e-af9a-10216921bf48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:13:42.095701 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:42.095662 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:13:42.096101 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:42.095714 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" podUID="45299d3c-52f3-4d0c-8dd3-af63948cf587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 18:13:42.379676 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:42.379597 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:13:42.379676 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:42.379658 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" podUID="6c7a6701-721d-448e-af9a-10216921bf48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:13:43.096318 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:43.096284 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:13:43.096834 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:43.096342 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" podUID="45299d3c-52f3-4d0c-8dd3-af63948cf587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 18:13:43.393590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:43.380886 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:13:43.393590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:43.380962 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" podUID="6c7a6701-721d-448e-af9a-10216921bf48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:13:44.096664 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:44.096304 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:13:44.096664 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:44.096357 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" podUID="45299d3c-52f3-4d0c-8dd3-af63948cf587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 18:13:44.378900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:44.378811 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:13:44.378900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:44.378870 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" podUID="6c7a6701-721d-448e-af9a-10216921bf48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:13:44.803416 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:44.803389 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9"] Oct 30 18:13:44.808738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:44.808717 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9" Oct 30 18:13:44.838850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:44.834390 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9"] Oct 30 18:13:44.848192 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:44.847144 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:13:44.848192 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:44.847319 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:13:44.976528 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:44.976485 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-704e738d-pssq9\" (UID: \"a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9" Oct 30 18:13:44.976681 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:44.976551 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94r6v\" (UniqueName: \"kubernetes.io/projected/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-kube-api-access-94r6v\") pod \"maas-api-revoke-sub-704e738d-pssq9\" (UID: \"a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9" Oct 30 18:13:44.976742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:44.976730 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-704e738d-pssq9\" (UID: \"a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9" Oct 30 18:13:45.077653 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:45.077548 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-704e738d-pssq9\" (UID: \"a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9" Oct 30 18:13:45.077653 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:45.077628 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-704e738d-pssq9\" (UID: \"a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9" Oct 30 18:13:45.077891 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:45.077652 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94r6v\" (UniqueName: \"kubernetes.io/projected/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-kube-api-access-94r6v\") pod \"maas-api-revoke-sub-704e738d-pssq9\" (UID: \"a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9" Oct 30 18:13:45.078641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:45.078611 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-704e738d-pssq9\" (UID: \"a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9" Oct 30 18:13:45.081091 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:45.080993 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-maas-api-xks-ca\") pod \"maas-api-revoke-sub-704e738d-pssq9\" (UID: \"a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9" Oct 30 18:13:45.095864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:45.095826 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:13:45.095961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:45.095890 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" podUID="45299d3c-52f3-4d0c-8dd3-af63948cf587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 18:13:45.096777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:45.096751 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94r6v\" (UniqueName: \"kubernetes.io/projected/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-kube-api-access-94r6v\") pod \"maas-api-revoke-sub-704e738d-pssq9\" (UID: \"a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9" Oct 30 18:13:45.187580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:45.187543 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9" Oct 30 18:13:45.382978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:45.380038 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:13:45.382978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:45.380084 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" podUID="6c7a6701-721d-448e-af9a-10216921bf48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:13:45.621940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:45.621902 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9"] Oct 30 18:13:45.641380 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:13:45.641197 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29a2b2a_cd25_4f9b_bfc0_bca1482cdb17.slice/crio-ea9537d8aca37f888e6cfce57a5739874daeeaff1992e3133e6c20f5117a0fec WatchSource:0}: Error finding container ea9537d8aca37f888e6cfce57a5739874daeeaff1992e3133e6c20f5117a0fec: Status 404 returned error can't find the container with id ea9537d8aca37f888e6cfce57a5739874daeeaff1992e3133e6c20f5117a0fec Oct 30 18:13:45.934593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:45.923197 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9" event={"ID":"a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17","Type":"ContainerStarted","Data":"ea9537d8aca37f888e6cfce57a5739874daeeaff1992e3133e6c20f5117a0fec"} Oct 30 18:13:46.095782 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.095745 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" start-of-body= Oct 30 18:13:46.095969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.095809 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" podUID="45299d3c-52f3-4d0c-8dd3-af63948cf587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": dial tcp 10.133.0.66:15021: connect: connection refused" Oct 30 18:13:46.273934 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.273895 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz"] Oct 30 18:13:46.278977 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.278950 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz" Oct 30 18:13:46.294098 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.294069 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz"] Oct 30 18:13:46.348890 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.348855 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c06fd47-8nwnz\" (UID: \"2fa09905-d3a9-42b9-9afe-3745e1bbaf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz" Oct 30 18:13:46.349089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.348978 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c06fd47-8nwnz\" (UID: \"2fa09905-d3a9-42b9-9afe-3745e1bbaf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz" Oct 30 18:13:46.349089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.349009 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khj44\" (UniqueName: \"kubernetes.io/projected/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-kube-api-access-khj44\") pod \"maas-api-revoke-sub-4c06fd47-8nwnz\" (UID: \"2fa09905-d3a9-42b9-9afe-3745e1bbaf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz" Oct 30 18:13:46.378892 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.378860 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" start-of-body= Oct 30 18:13:46.379033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.378914 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" podUID="6c7a6701-721d-448e-af9a-10216921bf48" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.67:15021/healthz/ready\": dial tcp 10.133.0.67:15021: connect: connection refused" Oct 30 18:13:46.450837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.450799 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c06fd47-8nwnz\" (UID: \"2fa09905-d3a9-42b9-9afe-3745e1bbaf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz" Oct 30 18:13:46.451046 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.450917 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c06fd47-8nwnz\" (UID: \"2fa09905-d3a9-42b9-9afe-3745e1bbaf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz" Oct 30 18:13:46.451046 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.450946 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khj44\" (UniqueName: \"kubernetes.io/projected/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-kube-api-access-khj44\") pod \"maas-api-revoke-sub-4c06fd47-8nwnz\" (UID: \"2fa09905-d3a9-42b9-9afe-3745e1bbaf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz" Oct 30 18:13:46.451820 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.451788 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c06fd47-8nwnz\" (UID: \"2fa09905-d3a9-42b9-9afe-3745e1bbaf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz" Oct 30 18:13:46.453491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.453470 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c06fd47-8nwnz\" (UID: \"2fa09905-d3a9-42b9-9afe-3745e1bbaf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz" Oct 30 18:13:46.469952 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.469924 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khj44\" (UniqueName: \"kubernetes.io/projected/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-kube-api-access-khj44\") pod \"maas-api-revoke-sub-4c06fd47-8nwnz\" (UID: \"2fa09905-d3a9-42b9-9afe-3745e1bbaf71\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz" Oct 30 18:13:46.603057 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.602949 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz" Oct 30 18:13:46.927208 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.924810 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz"] Oct 30 18:13:46.929177 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:13:46.929147 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fa09905_d3a9_42b9_9afe_3745e1bbaf71.slice/crio-9cb0640c8b5ef992266b257692ffe285d52f88a37f28ce3ac82756156bb288e2 WatchSource:0}: Error finding container 9cb0640c8b5ef992266b257692ffe285d52f88a37f28ce3ac82756156bb288e2: Status 404 returned error can't find the container with id 9cb0640c8b5ef992266b257692ffe285d52f88a37f28ce3ac82756156bb288e2 Oct 30 18:13:46.933472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.933445 5232 generic.go:356] "Generic (PLEG): container finished" podID="a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17" containerID="7e5d0e7be3a7a2f28b205921fc4b9806ac6db68d0544fcf0831d85369d548c17" exitCode=0 Oct 30 18:13:46.933582 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:46.933535 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9" event={"ID":"a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17","Type":"ContainerDied","Data":"7e5d0e7be3a7a2f28b205921fc4b9806ac6db68d0544fcf0831d85369d548c17"} Oct 30 18:13:47.100666 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:47.100630 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:47.100966 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:47.100945 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:47.101901 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:47.101880 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:13:47.383106 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:47.383050 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:47.383459 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:47.383352 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:47.384021 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:47.384003 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:13:47.941681 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:47.941647 5232 generic.go:356] "Generic (PLEG): container finished" podID="2fa09905-d3a9-42b9-9afe-3745e1bbaf71" containerID="d2e8532a44453206f628fd71268e51d1df128e7053c1a512368c0c57391925a2" exitCode=0 Oct 30 18:13:47.941880 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:47.941741 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz" event={"ID":"2fa09905-d3a9-42b9-9afe-3745e1bbaf71","Type":"ContainerDied","Data":"d2e8532a44453206f628fd71268e51d1df128e7053c1a512368c0c57391925a2"} Oct 30 18:13:47.941880 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:47.941804 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz" event={"ID":"2fa09905-d3a9-42b9-9afe-3745e1bbaf71","Type":"ContainerStarted","Data":"9cb0640c8b5ef992266b257692ffe285d52f88a37f28ce3ac82756156bb288e2"} Oct 30 18:13:48.095737 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:48.095710 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9" Oct 30 18:13:48.177841 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:48.177803 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-maas-api-openshift-ca\") pod \"a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17\" (UID: \"a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17\") " Oct 30 18:13:48.178040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:48.177868 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-kube-api-access-94r6v\" (UniqueName: \"kubernetes.io/projected/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-kube-api-access-94r6v\") pod \"a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17\" (UID: \"a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17\") " Oct 30 18:13:48.178040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:48.177911 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-maas-api-xks-ca\") pod \"a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17\" (UID: \"a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17\") " Oct 30 18:13:48.178304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:48.178275 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-maas-api-openshift-ca" pod "a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17" (UID: "a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:48.178415 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:48.178399 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:48.180132 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:48.180108 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-maas-api-xks-ca" pod "a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17" (UID: "a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:48.180259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:48.180239 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-kube-api-access-94r6v" pod "a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17" (UID: "a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17"). InnerVolumeSpecName "kube-api-access-94r6v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:48.279302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:48.279270 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94r6v\" (UniqueName: \"kubernetes.io/projected/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-kube-api-access-94r6v\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:48.279302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:48.279298 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:48.950830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:48.950798 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9" Oct 30 18:13:48.951325 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:48.950858 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9" event={"ID":"a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17","Type":"ContainerDied","Data":"ea9537d8aca37f888e6cfce57a5739874daeeaff1992e3133e6c20f5117a0fec"} Oct 30 18:13:48.951325 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:48.950904 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ea9537d8aca37f888e6cfce57a5739874daeeaff1992e3133e6c20f5117a0fec" Oct 30 18:13:49.110846 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:49.110824 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz" Oct 30 18:13:49.193631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:49.193601 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-maas-api-xks-ca\") pod \"2fa09905-d3a9-42b9-9afe-3745e1bbaf71\" (UID: \"2fa09905-d3a9-42b9-9afe-3745e1bbaf71\") " Oct 30 18:13:49.193794 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:49.193657 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-kube-api-access-khj44\" (UniqueName: \"kubernetes.io/projected/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-kube-api-access-khj44\") pod \"2fa09905-d3a9-42b9-9afe-3745e1bbaf71\" (UID: \"2fa09905-d3a9-42b9-9afe-3745e1bbaf71\") " Oct 30 18:13:49.193794 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:49.193699 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-maas-api-openshift-ca\") pod \"2fa09905-d3a9-42b9-9afe-3745e1bbaf71\" (UID: \"2fa09905-d3a9-42b9-9afe-3745e1bbaf71\") " Oct 30 18:13:49.194118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:49.194081 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-maas-api-openshift-ca" pod "2fa09905-d3a9-42b9-9afe-3745e1bbaf71" (UID: "2fa09905-d3a9-42b9-9afe-3745e1bbaf71"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:49.195895 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:49.195873 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-maas-api-xks-ca" pod "2fa09905-d3a9-42b9-9afe-3745e1bbaf71" (UID: "2fa09905-d3a9-42b9-9afe-3745e1bbaf71"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:49.196037 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:49.196013 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-kube-api-access-khj44" pod "2fa09905-d3a9-42b9-9afe-3745e1bbaf71" (UID: "2fa09905-d3a9-42b9-9afe-3745e1bbaf71"). InnerVolumeSpecName "kube-api-access-khj44". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:49.295236 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:49.295190 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:49.295236 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:49.295234 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khj44\" (UniqueName: \"kubernetes.io/projected/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-kube-api-access-khj44\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:49.295236 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:49.295249 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2fa09905-d3a9-42b9-9afe-3745e1bbaf71-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:49.958488 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:49.958451 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz" event={"ID":"2fa09905-d3a9-42b9-9afe-3745e1bbaf71","Type":"ContainerDied","Data":"9cb0640c8b5ef992266b257692ffe285d52f88a37f28ce3ac82756156bb288e2"} Oct 30 18:13:49.958899 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:49.958522 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9cb0640c8b5ef992266b257692ffe285d52f88a37f28ce3ac82756156bb288e2" Oct 30 18:13:49.958899 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:49.958461 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz" Oct 30 18:13:50.458929 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.458892 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb"] Oct 30 18:13:50.459413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.459400 5232 cpu_manager.go:414] "containerMap: removing container" podUID="2fa09905-d3a9-42b9-9afe-3745e1bbaf71" containerName="revoke-keys" Oct 30 18:13:50.459455 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.459415 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fa09905-d3a9-42b9-9afe-3745e1bbaf71" containerName="revoke-keys" Oct 30 18:13:50.459455 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.459448 5232 cpu_manager.go:414] "containerMap: removing container" podUID="a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17" containerName="revoke-keys" Oct 30 18:13:50.459455 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.459453 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17" containerName="revoke-keys" Oct 30 18:13:50.459607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.459553 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17" containerName="revoke-keys" Oct 30 18:13:50.459607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.459565 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fa09905-d3a9-42b9-9afe-3745e1bbaf71" containerName="revoke-keys" Oct 30 18:13:50.463139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.463095 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb" Oct 30 18:13:50.491437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.491095 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:13:50.513647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.503925 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:13:50.523161 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.523128 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb"] Oct 30 18:13:50.532116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.532074 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2459w\" (UniqueName: \"kubernetes.io/projected/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-kube-api-access-2459w\") pod \"maas-api-revoke-sub-559a064f-f47wb\" (UID: \"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb" Oct 30 18:13:50.532330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.532169 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-559a064f-f47wb\" (UID: \"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb" Oct 30 18:13:50.532416 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.532381 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-559a064f-f47wb\" (UID: \"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb" Oct 30 18:13:50.633905 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.633815 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-559a064f-f47wb\" (UID: \"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb" Oct 30 18:13:50.633905 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.633878 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2459w\" (UniqueName: \"kubernetes.io/projected/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-kube-api-access-2459w\") pod \"maas-api-revoke-sub-559a064f-f47wb\" (UID: \"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb" Oct 30 18:13:50.634137 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.633927 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-559a064f-f47wb\" (UID: \"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb" Oct 30 18:13:50.634475 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.634454 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-559a064f-f47wb\" (UID: \"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb" Oct 30 18:13:50.636440 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.636422 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-559a064f-f47wb\" (UID: \"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb" Oct 30 18:13:50.652770 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.652740 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2459w\" (UniqueName: \"kubernetes.io/projected/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-kube-api-access-2459w\") pod \"maas-api-revoke-sub-559a064f-f47wb\" (UID: \"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb" Oct 30 18:13:50.796346 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:50.796308 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb" Oct 30 18:13:51.097059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:51.097004 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb"] Oct 30 18:13:51.099873 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:13:51.099847 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4790a52_fcfc_4d32_becc_61ff1eb5ddd3.slice/crio-054b563f8f84bec02ac7cd63b0f70d276080f2baff7e69eaf3ecebe4804a0587 WatchSource:0}: Error finding container 054b563f8f84bec02ac7cd63b0f70d276080f2baff7e69eaf3ecebe4804a0587: Status 404 returned error can't find the container with id 054b563f8f84bec02ac7cd63b0f70d276080f2baff7e69eaf3ecebe4804a0587 Oct 30 18:13:51.972789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:51.972690 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb" event={"ID":"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3","Type":"ContainerStarted","Data":"860e0f4b112b9b2f06d5c7726f9fa8ea2f24c6937fd92c6cb736db557e4e47c2"} Oct 30 18:13:51.972789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:51.972751 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb" event={"ID":"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3","Type":"ContainerStarted","Data":"054b563f8f84bec02ac7cd63b0f70d276080f2baff7e69eaf3ecebe4804a0587"} Oct 30 18:13:52.011303 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:52.011233 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb" podStartSLOduration=2.01121397 podStartE2EDuration="2.01121397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:52.009713663 +0000 UTC m=+731.326133981" watchObservedRunningTime="2026-10-30 18:13:52.01121397 +0000 UTC m=+731.327634292" Oct 30 18:13:52.128887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:52.128825 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2"] Oct 30 18:13:52.162033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:52.161998 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2"] Oct 30 18:13:52.162157 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:52.162125 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2" Oct 30 18:13:52.255170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:52.255133 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c44655c7-466e-419a-a2b4-04d6f11df1d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07c65b1a-tppn2\" (UID: \"c44655c7-466e-419a-a2b4-04d6f11df1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2" Oct 30 18:13:52.255365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:52.255278 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtjvk\" (UniqueName: \"kubernetes.io/projected/c44655c7-466e-419a-a2b4-04d6f11df1d6-kube-api-access-mtjvk\") pod \"maas-api-revoke-sub-07c65b1a-tppn2\" (UID: \"c44655c7-466e-419a-a2b4-04d6f11df1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2" Oct 30 18:13:52.255440 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:52.255418 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c44655c7-466e-419a-a2b4-04d6f11df1d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07c65b1a-tppn2\" (UID: \"c44655c7-466e-419a-a2b4-04d6f11df1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2" Oct 30 18:13:52.357142 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:52.357103 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c44655c7-466e-419a-a2b4-04d6f11df1d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07c65b1a-tppn2\" (UID: \"c44655c7-466e-419a-a2b4-04d6f11df1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2" Oct 30 18:13:52.357353 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:52.357187 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtjvk\" (UniqueName: \"kubernetes.io/projected/c44655c7-466e-419a-a2b4-04d6f11df1d6-kube-api-access-mtjvk\") pod \"maas-api-revoke-sub-07c65b1a-tppn2\" (UID: \"c44655c7-466e-419a-a2b4-04d6f11df1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2" Oct 30 18:13:52.357353 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:52.357241 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c44655c7-466e-419a-a2b4-04d6f11df1d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07c65b1a-tppn2\" (UID: \"c44655c7-466e-419a-a2b4-04d6f11df1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2" Oct 30 18:13:52.357872 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:52.357845 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c44655c7-466e-419a-a2b4-04d6f11df1d6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07c65b1a-tppn2\" (UID: \"c44655c7-466e-419a-a2b4-04d6f11df1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2" Oct 30 18:13:52.359789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:52.359766 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c44655c7-466e-419a-a2b4-04d6f11df1d6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07c65b1a-tppn2\" (UID: \"c44655c7-466e-419a-a2b4-04d6f11df1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2" Oct 30 18:13:52.376246 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:52.376217 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtjvk\" (UniqueName: \"kubernetes.io/projected/c44655c7-466e-419a-a2b4-04d6f11df1d6-kube-api-access-mtjvk\") pod \"maas-api-revoke-sub-07c65b1a-tppn2\" (UID: \"c44655c7-466e-419a-a2b4-04d6f11df1d6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2" Oct 30 18:13:52.522584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:52.522463 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2" Oct 30 18:13:52.819001 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:52.818976 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2"] Oct 30 18:13:52.821295 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:13:52.821265 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc44655c7_466e_419a_a2b4_04d6f11df1d6.slice/crio-2abc61a9957de5d3c8fd06b71489698beba5bee29ea442659ce8f3ac4137bfd1 WatchSource:0}: Error finding container 2abc61a9957de5d3c8fd06b71489698beba5bee29ea442659ce8f3ac4137bfd1: Status 404 returned error can't find the container with id 2abc61a9957de5d3c8fd06b71489698beba5bee29ea442659ce8f3ac4137bfd1 Oct 30 18:13:52.980959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:52.980924 5232 generic.go:356] "Generic (PLEG): container finished" podID="a4790a52-fcfc-4d32-becc-61ff1eb5ddd3" containerID="860e0f4b112b9b2f06d5c7726f9fa8ea2f24c6937fd92c6cb736db557e4e47c2" exitCode=0 Oct 30 18:13:52.981120 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:52.981012 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb" event={"ID":"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3","Type":"ContainerDied","Data":"860e0f4b112b9b2f06d5c7726f9fa8ea2f24c6937fd92c6cb736db557e4e47c2"} Oct 30 18:13:52.982326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:52.982305 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2" event={"ID":"c44655c7-466e-419a-a2b4-04d6f11df1d6","Type":"ContainerStarted","Data":"2abc61a9957de5d3c8fd06b71489698beba5bee29ea442659ce8f3ac4137bfd1"} Oct 30 18:13:53.990569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:53.990539 5232 generic.go:356] "Generic (PLEG): container finished" podID="c44655c7-466e-419a-a2b4-04d6f11df1d6" containerID="baeebd6b2fc27e7f514f199cdc28d397972e304bb3ad6db19eca64dac327f085" exitCode=0 Oct 30 18:13:53.990948 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:53.990607 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2" event={"ID":"c44655c7-466e-419a-a2b4-04d6f11df1d6","Type":"ContainerDied","Data":"baeebd6b2fc27e7f514f199cdc28d397972e304bb3ad6db19eca64dac327f085"} Oct 30 18:13:54.159535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:54.159479 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb" Oct 30 18:13:54.285432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:54.285345 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-kube-api-access-2459w\" (UniqueName: \"kubernetes.io/projected/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-kube-api-access-2459w\") pod \"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3\" (UID: \"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3\") " Oct 30 18:13:54.285432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:54.285390 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-maas-api-xks-ca\") pod \"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3\" (UID: \"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3\") " Oct 30 18:13:54.285662 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:54.285435 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-maas-api-openshift-ca\") pod \"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3\" (UID: \"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3\") " Oct 30 18:13:54.285897 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:54.285867 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-maas-api-openshift-ca" pod "a4790a52-fcfc-4d32-becc-61ff1eb5ddd3" (UID: "a4790a52-fcfc-4d32-becc-61ff1eb5ddd3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:54.287777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:54.287753 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-maas-api-xks-ca" pod "a4790a52-fcfc-4d32-becc-61ff1eb5ddd3" (UID: "a4790a52-fcfc-4d32-becc-61ff1eb5ddd3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:54.287887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:54.287858 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-kube-api-access-2459w" pod "a4790a52-fcfc-4d32-becc-61ff1eb5ddd3" (UID: "a4790a52-fcfc-4d32-becc-61ff1eb5ddd3"). InnerVolumeSpecName "kube-api-access-2459w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:54.386725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:54.386692 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2459w\" (UniqueName: \"kubernetes.io/projected/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-kube-api-access-2459w\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:54.386725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:54.386721 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:54.386725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:54.386730 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:54.998922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:54.998886 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb" event={"ID":"a4790a52-fcfc-4d32-becc-61ff1eb5ddd3","Type":"ContainerDied","Data":"054b563f8f84bec02ac7cd63b0f70d276080f2baff7e69eaf3ecebe4804a0587"} Oct 30 18:13:54.999439 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:54.998940 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="054b563f8f84bec02ac7cd63b0f70d276080f2baff7e69eaf3ecebe4804a0587" Oct 30 18:13:54.999439 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:54.999005 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb" Oct 30 18:13:55.148716 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:55.148694 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2" Oct 30 18:13:55.296325 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:55.296285 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c44655c7-466e-419a-a2b4-04d6f11df1d6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c44655c7-466e-419a-a2b4-04d6f11df1d6-maas-api-openshift-ca\") pod \"c44655c7-466e-419a-a2b4-04d6f11df1d6\" (UID: \"c44655c7-466e-419a-a2b4-04d6f11df1d6\") " Oct 30 18:13:55.296602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:55.296339 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c44655c7-466e-419a-a2b4-04d6f11df1d6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c44655c7-466e-419a-a2b4-04d6f11df1d6-maas-api-xks-ca\") pod \"c44655c7-466e-419a-a2b4-04d6f11df1d6\" (UID: \"c44655c7-466e-419a-a2b4-04d6f11df1d6\") " Oct 30 18:13:55.296602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:55.296463 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c44655c7-466e-419a-a2b4-04d6f11df1d6-kube-api-access-mtjvk\" (UniqueName: \"kubernetes.io/projected/c44655c7-466e-419a-a2b4-04d6f11df1d6-kube-api-access-mtjvk\") pod \"c44655c7-466e-419a-a2b4-04d6f11df1d6\" (UID: \"c44655c7-466e-419a-a2b4-04d6f11df1d6\") " Oct 30 18:13:55.296760 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:55.296679 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c44655c7-466e-419a-a2b4-04d6f11df1d6-maas-api-openshift-ca" pod "c44655c7-466e-419a-a2b4-04d6f11df1d6" (UID: "c44655c7-466e-419a-a2b4-04d6f11df1d6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:13:55.297012 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:55.296986 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c44655c7-466e-419a-a2b4-04d6f11df1d6-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:55.298694 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:55.298671 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c44655c7-466e-419a-a2b4-04d6f11df1d6-maas-api-xks-ca" pod "c44655c7-466e-419a-a2b4-04d6f11df1d6" (UID: "c44655c7-466e-419a-a2b4-04d6f11df1d6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:13:55.298801 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:55.298744 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c44655c7-466e-419a-a2b4-04d6f11df1d6-kube-api-access-mtjvk" pod "c44655c7-466e-419a-a2b4-04d6f11df1d6" (UID: "c44655c7-466e-419a-a2b4-04d6f11df1d6"). InnerVolumeSpecName "kube-api-access-mtjvk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:13:55.400584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:55.398168 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtjvk\" (UniqueName: \"kubernetes.io/projected/c44655c7-466e-419a-a2b4-04d6f11df1d6-kube-api-access-mtjvk\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:55.400584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:55.398209 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c44655c7-466e-419a-a2b4-04d6f11df1d6-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:13:56.006873 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:56.006842 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2" Oct 30 18:13:56.007306 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:56.006884 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2" event={"ID":"c44655c7-466e-419a-a2b4-04d6f11df1d6","Type":"ContainerDied","Data":"2abc61a9957de5d3c8fd06b71489698beba5bee29ea442659ce8f3ac4137bfd1"} Oct 30 18:13:56.007306 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:56.006942 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2abc61a9957de5d3c8fd06b71489698beba5bee29ea442659ce8f3ac4137bfd1" Oct 30 18:13:57.085490 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:13:57.085452 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:13:58.008575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.008535 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp"] Oct 30 18:13:58.009124 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.009111 5232 cpu_manager.go:414] "containerMap: removing container" podUID="a4790a52-fcfc-4d32-becc-61ff1eb5ddd3" containerName="revoke-keys" Oct 30 18:13:58.009164 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.009126 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4790a52-fcfc-4d32-becc-61ff1eb5ddd3" containerName="revoke-keys" Oct 30 18:13:58.009164 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.009138 5232 cpu_manager.go:414] "containerMap: removing container" podUID="c44655c7-466e-419a-a2b4-04d6f11df1d6" containerName="revoke-keys" Oct 30 18:13:58.009164 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.009143 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c44655c7-466e-419a-a2b4-04d6f11df1d6" containerName="revoke-keys" Oct 30 18:13:58.009244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.009228 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4790a52-fcfc-4d32-becc-61ff1eb5ddd3" containerName="revoke-keys" Oct 30 18:13:58.009244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.009236 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="c44655c7-466e-419a-a2b4-04d6f11df1d6" containerName="revoke-keys" Oct 30 18:13:58.012748 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.012726 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp" Oct 30 18:13:58.032303 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.032273 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp"] Oct 30 18:13:58.041642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.039618 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:13:58.041642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.039667 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:13:58.136202 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.136171 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a474cc-27db-44de-a436-e9d490d78ec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-355285a5-xhcrp\" (UID: \"a0a474cc-27db-44de-a436-e9d490d78ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp" Oct 30 18:13:58.136603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.136229 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a474cc-27db-44de-a436-e9d490d78ec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-355285a5-xhcrp\" (UID: \"a0a474cc-27db-44de-a436-e9d490d78ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp" Oct 30 18:13:58.136603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.136354 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xl5p5\" (UniqueName: \"kubernetes.io/projected/a0a474cc-27db-44de-a436-e9d490d78ec1-kube-api-access-xl5p5\") pod \"maas-api-revoke-sub-355285a5-xhcrp\" (UID: \"a0a474cc-27db-44de-a436-e9d490d78ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp" Oct 30 18:13:58.237823 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.237779 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xl5p5\" (UniqueName: \"kubernetes.io/projected/a0a474cc-27db-44de-a436-e9d490d78ec1-kube-api-access-xl5p5\") pod \"maas-api-revoke-sub-355285a5-xhcrp\" (UID: \"a0a474cc-27db-44de-a436-e9d490d78ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp" Oct 30 18:13:58.237999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.237933 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a474cc-27db-44de-a436-e9d490d78ec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-355285a5-xhcrp\" (UID: \"a0a474cc-27db-44de-a436-e9d490d78ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp" Oct 30 18:13:58.237999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.237976 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a474cc-27db-44de-a436-e9d490d78ec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-355285a5-xhcrp\" (UID: \"a0a474cc-27db-44de-a436-e9d490d78ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp" Oct 30 18:13:58.238685 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.238660 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a474cc-27db-44de-a436-e9d490d78ec1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-355285a5-xhcrp\" (UID: \"a0a474cc-27db-44de-a436-e9d490d78ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp" Oct 30 18:13:58.240385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.240367 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a474cc-27db-44de-a436-e9d490d78ec1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-355285a5-xhcrp\" (UID: \"a0a474cc-27db-44de-a436-e9d490d78ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp" Oct 30 18:13:58.256994 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.256954 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xl5p5\" (UniqueName: \"kubernetes.io/projected/a0a474cc-27db-44de-a436-e9d490d78ec1-kube-api-access-xl5p5\") pod \"maas-api-revoke-sub-355285a5-xhcrp\" (UID: \"a0a474cc-27db-44de-a436-e9d490d78ec1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp" Oct 30 18:13:58.327032 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.326957 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp" Oct 30 18:13:58.613994 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.613915 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms"] Oct 30 18:13:58.619177 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.619153 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms" Oct 30 18:13:58.629610 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.629588 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp"] Oct 30 18:13:58.630930 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:13:58.630835 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0a474cc_27db_44de_a436_e9d490d78ec1.slice/crio-04c81f5cdf114b70ba821859f3cc9edb8046d9b66d4d3016788ff31b16729606 WatchSource:0}: Error finding container 04c81f5cdf114b70ba821859f3cc9edb8046d9b66d4d3016788ff31b16729606: Status 404 returned error can't find the container with id 04c81f5cdf114b70ba821859f3cc9edb8046d9b66d4d3016788ff31b16729606 Oct 30 18:13:58.636674 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.636651 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms"] Oct 30 18:13:58.758542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.758238 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7be05604-c569-49a4-9538-238d1728e104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8b7933b-7qcms\" (UID: \"7be05604-c569-49a4-9538-238d1728e104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms" Oct 30 18:13:58.758542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.758308 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbs79\" (UniqueName: \"kubernetes.io/projected/7be05604-c569-49a4-9538-238d1728e104-kube-api-access-kbs79\") pod \"maas-api-revoke-sub-a8b7933b-7qcms\" (UID: \"7be05604-c569-49a4-9538-238d1728e104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms" Oct 30 18:13:58.758793 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.758597 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7be05604-c569-49a4-9538-238d1728e104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8b7933b-7qcms\" (UID: \"7be05604-c569-49a4-9538-238d1728e104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms" Oct 30 18:13:58.860501 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.860466 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7be05604-c569-49a4-9538-238d1728e104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8b7933b-7qcms\" (UID: \"7be05604-c569-49a4-9538-238d1728e104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms" Oct 30 18:13:58.860674 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.860627 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7be05604-c569-49a4-9538-238d1728e104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8b7933b-7qcms\" (UID: \"7be05604-c569-49a4-9538-238d1728e104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms" Oct 30 18:13:58.860674 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.860668 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbs79\" (UniqueName: \"kubernetes.io/projected/7be05604-c569-49a4-9538-238d1728e104-kube-api-access-kbs79\") pod \"maas-api-revoke-sub-a8b7933b-7qcms\" (UID: \"7be05604-c569-49a4-9538-238d1728e104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms" Oct 30 18:13:58.861228 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.861206 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7be05604-c569-49a4-9538-238d1728e104-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a8b7933b-7qcms\" (UID: \"7be05604-c569-49a4-9538-238d1728e104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms" Oct 30 18:13:58.863242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.863218 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7be05604-c569-49a4-9538-238d1728e104-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a8b7933b-7qcms\" (UID: \"7be05604-c569-49a4-9538-238d1728e104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms" Oct 30 18:13:58.879898 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.879831 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbs79\" (UniqueName: \"kubernetes.io/projected/7be05604-c569-49a4-9538-238d1728e104-kube-api-access-kbs79\") pod \"maas-api-revoke-sub-a8b7933b-7qcms\" (UID: \"7be05604-c569-49a4-9538-238d1728e104\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms" Oct 30 18:13:58.950383 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:58.950355 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms" Oct 30 18:13:59.068245 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:59.068209 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp" event={"ID":"a0a474cc-27db-44de-a436-e9d490d78ec1","Type":"ContainerStarted","Data":"21c10db08f62e7afbe59732832398cfe40579dbd24d6ac270195e5c0ca7ac0c7"} Oct 30 18:13:59.068397 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:59.068270 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp" event={"ID":"a0a474cc-27db-44de-a436-e9d490d78ec1","Type":"ContainerStarted","Data":"04c81f5cdf114b70ba821859f3cc9edb8046d9b66d4d3016788ff31b16729606"} Oct 30 18:13:59.107400 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:59.107341 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp" podStartSLOduration=2.107325917 podStartE2EDuration="2.107325917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:13:59.105327295 +0000 UTC m=+738.421747610" watchObservedRunningTime="2026-10-30 18:13:59.107325917 +0000 UTC m=+738.423746228" Oct 30 18:13:59.247165 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:13:59.247140 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms"] Oct 30 18:13:59.249305 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:13:59.249275 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7be05604_c569_49a4_9538_238d1728e104.slice/crio-4f977777029960742b4fd6ebc3663a965709e60caa28c155cfd19e9fb0c3671c WatchSource:0}: Error finding container 4f977777029960742b4fd6ebc3663a965709e60caa28c155cfd19e9fb0c3671c: Status 404 returned error can't find the container with id 4f977777029960742b4fd6ebc3663a965709e60caa28c155cfd19e9fb0c3671c Oct 30 18:14:00.083387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:00.083350 5232 generic.go:356] "Generic (PLEG): container finished" podID="a0a474cc-27db-44de-a436-e9d490d78ec1" containerID="21c10db08f62e7afbe59732832398cfe40579dbd24d6ac270195e5c0ca7ac0c7" exitCode=0 Oct 30 18:14:00.083631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:00.083587 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp" event={"ID":"a0a474cc-27db-44de-a436-e9d490d78ec1","Type":"ContainerDied","Data":"21c10db08f62e7afbe59732832398cfe40579dbd24d6ac270195e5c0ca7ac0c7"} Oct 30 18:14:00.085569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:00.085540 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms" event={"ID":"7be05604-c569-49a4-9538-238d1728e104","Type":"ContainerStarted","Data":"d3296c3e674c97821abb98a2da29084860da2cee9349c8870f1d91175661d29a"} Oct 30 18:14:00.085703 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:00.085590 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms" event={"ID":"7be05604-c569-49a4-9538-238d1728e104","Type":"ContainerStarted","Data":"4f977777029960742b4fd6ebc3663a965709e60caa28c155cfd19e9fb0c3671c"} Oct 30 18:14:00.290586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:00.272150 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms" podStartSLOduration=2.272126778 podStartE2EDuration="2.272126778s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:13:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:00.240262371 +0000 UTC m=+739.556682693" watchObservedRunningTime="2026-10-30 18:14:00.272126778 +0000 UTC m=+739.588547100" Oct 30 18:14:01.184133 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:01.184093 5232 generic.go:356] "Generic (PLEG): container finished" podID="7be05604-c569-49a4-9538-238d1728e104" containerID="d3296c3e674c97821abb98a2da29084860da2cee9349c8870f1d91175661d29a" exitCode=0 Oct 30 18:14:01.185161 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:01.185129 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms" event={"ID":"7be05604-c569-49a4-9538-238d1728e104","Type":"ContainerDied","Data":"d3296c3e674c97821abb98a2da29084860da2cee9349c8870f1d91175661d29a"} Oct 30 18:14:01.996741 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:01.996709 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp" Oct 30 18:14:02.058537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.045901 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0a474cc-27db-44de-a436-e9d490d78ec1-kube-api-access-xl5p5\" (UniqueName: \"kubernetes.io/projected/a0a474cc-27db-44de-a436-e9d490d78ec1-kube-api-access-xl5p5\") pod \"a0a474cc-27db-44de-a436-e9d490d78ec1\" (UID: \"a0a474cc-27db-44de-a436-e9d490d78ec1\") " Oct 30 18:14:02.058537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.045963 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0a474cc-27db-44de-a436-e9d490d78ec1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a474cc-27db-44de-a436-e9d490d78ec1-maas-api-openshift-ca\") pod \"a0a474cc-27db-44de-a436-e9d490d78ec1\" (UID: \"a0a474cc-27db-44de-a436-e9d490d78ec1\") " Oct 30 18:14:02.058537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.046160 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0a474cc-27db-44de-a436-e9d490d78ec1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a474cc-27db-44de-a436-e9d490d78ec1-maas-api-xks-ca\") pod \"a0a474cc-27db-44de-a436-e9d490d78ec1\" (UID: \"a0a474cc-27db-44de-a436-e9d490d78ec1\") " Oct 30 18:14:02.058537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.048930 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0a474cc-27db-44de-a436-e9d490d78ec1-maas-api-openshift-ca" pod "a0a474cc-27db-44de-a436-e9d490d78ec1" (UID: "a0a474cc-27db-44de-a436-e9d490d78ec1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:02.162577 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.155679 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0a474cc-27db-44de-a436-e9d490d78ec1-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:14:02.176237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.176192 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0a474cc-27db-44de-a436-e9d490d78ec1-maas-api-xks-ca" pod "a0a474cc-27db-44de-a436-e9d490d78ec1" (UID: "a0a474cc-27db-44de-a436-e9d490d78ec1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:02.196595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.185213 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0a474cc-27db-44de-a436-e9d490d78ec1-kube-api-access-xl5p5" pod "a0a474cc-27db-44de-a436-e9d490d78ec1" (UID: "a0a474cc-27db-44de-a436-e9d490d78ec1"). InnerVolumeSpecName "kube-api-access-xl5p5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:02.273071 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.273034 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xl5p5\" (UniqueName: \"kubernetes.io/projected/a0a474cc-27db-44de-a436-e9d490d78ec1-kube-api-access-xl5p5\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:14:02.280537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.279029 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0a474cc-27db-44de-a436-e9d490d78ec1-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:14:02.281538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.280940 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp" Oct 30 18:14:02.282523 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.281641 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp" event={"ID":"a0a474cc-27db-44de-a436-e9d490d78ec1","Type":"ContainerDied","Data":"04c81f5cdf114b70ba821859f3cc9edb8046d9b66d4d3016788ff31b16729606"} Oct 30 18:14:02.282523 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.281790 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04c81f5cdf114b70ba821859f3cc9edb8046d9b66d4d3016788ff31b16729606" Oct 30 18:14:02.476670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.476632 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms" Oct 30 18:14:02.482166 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.482138 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7be05604-c569-49a4-9538-238d1728e104-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7be05604-c569-49a4-9538-238d1728e104-maas-api-openshift-ca\") pod \"7be05604-c569-49a4-9538-238d1728e104\" (UID: \"7be05604-c569-49a4-9538-238d1728e104\") " Oct 30 18:14:02.482303 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.482265 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7be05604-c569-49a4-9538-238d1728e104-kube-api-access-kbs79\" (UniqueName: \"kubernetes.io/projected/7be05604-c569-49a4-9538-238d1728e104-kube-api-access-kbs79\") pod \"7be05604-c569-49a4-9538-238d1728e104\" (UID: \"7be05604-c569-49a4-9538-238d1728e104\") " Oct 30 18:14:02.482346 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.482337 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7be05604-c569-49a4-9538-238d1728e104-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7be05604-c569-49a4-9538-238d1728e104-maas-api-xks-ca\") pod \"7be05604-c569-49a4-9538-238d1728e104\" (UID: \"7be05604-c569-49a4-9538-238d1728e104\") " Oct 30 18:14:02.482591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.482564 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7be05604-c569-49a4-9538-238d1728e104-maas-api-openshift-ca" pod "7be05604-c569-49a4-9538-238d1728e104" (UID: "7be05604-c569-49a4-9538-238d1728e104"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:02.482869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.482850 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7be05604-c569-49a4-9538-238d1728e104-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:14:02.484562 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.484497 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7be05604-c569-49a4-9538-238d1728e104-kube-api-access-kbs79" pod "7be05604-c569-49a4-9538-238d1728e104" (UID: "7be05604-c569-49a4-9538-238d1728e104"). InnerVolumeSpecName "kube-api-access-kbs79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:02.484727 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.484703 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7be05604-c569-49a4-9538-238d1728e104-maas-api-xks-ca" pod "7be05604-c569-49a4-9538-238d1728e104" (UID: "7be05604-c569-49a4-9538-238d1728e104"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:02.584106 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.584069 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kbs79\" (UniqueName: \"kubernetes.io/projected/7be05604-c569-49a4-9538-238d1728e104-kube-api-access-kbs79\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:14:02.584106 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:02.584106 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7be05604-c569-49a4-9538-238d1728e104-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:14:03.287869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:03.287838 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms" event={"ID":"7be05604-c569-49a4-9538-238d1728e104","Type":"ContainerDied","Data":"4f977777029960742b4fd6ebc3663a965709e60caa28c155cfd19e9fb0c3671c"} Oct 30 18:14:03.287869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:03.287868 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms" Oct 30 18:14:03.288278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:03.287891 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f977777029960742b4fd6ebc3663a965709e60caa28c155cfd19e9fb0c3671c" Oct 30 18:14:06.149468 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.149436 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm"] Oct 30 18:14:06.150244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.150223 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7be05604-c569-49a4-9538-238d1728e104" containerName="revoke-keys" Oct 30 18:14:06.150368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.150246 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7be05604-c569-49a4-9538-238d1728e104" containerName="revoke-keys" Oct 30 18:14:06.150368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.150276 5232 cpu_manager.go:414] "containerMap: removing container" podUID="a0a474cc-27db-44de-a436-e9d490d78ec1" containerName="revoke-keys" Oct 30 18:14:06.150368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.150285 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0a474cc-27db-44de-a436-e9d490d78ec1" containerName="revoke-keys" Oct 30 18:14:06.150549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.150419 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7be05604-c569-49a4-9538-238d1728e104" containerName="revoke-keys" Oct 30 18:14:06.150549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.150435 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0a474cc-27db-44de-a436-e9d490d78ec1" containerName="revoke-keys" Oct 30 18:14:06.154040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.154019 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.170655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.170630 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm"] Oct 30 18:14:06.179168 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.179142 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-4ef6ac\"/\"distinct-w4-4ef6ac-kserve-self-signed-certs\"" Oct 30 18:14:06.272262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.272223 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q4cbt\" (UniqueName: \"kubernetes.io/projected/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-kube-api-access-q4cbt\") pod \"distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.272262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.272273 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-tmp-dir\") pod \"distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.272536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.272466 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-dshm\") pod \"distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.272613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.272593 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-model-cache\") pod \"distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.272651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.272630 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-tls-certs\") pod \"distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.272710 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.272696 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-home\") pod \"distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.380460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.380410 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-dshm\") pod \"distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.381255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.381222 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-model-cache\") pod \"distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.381654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.381622 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-tls-certs\") pod \"distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.381859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.381838 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-model-cache\") pod \"distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.387540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.382096 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-home\") pod \"distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.387540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.382181 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q4cbt\" (UniqueName: \"kubernetes.io/projected/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-kube-api-access-q4cbt\") pod \"distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.387540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.382224 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-tmp-dir\") pod \"distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.387540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.382664 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-tmp-dir\") pod \"distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.387540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.382886 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-home\") pod \"distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.410564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.410468 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-dshm\") pod \"distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.412214 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.412183 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-tls-certs\") pod \"distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.415544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.415205 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q4cbt\" (UniqueName: \"kubernetes.io/projected/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-kube-api-access-q4cbt\") pod \"distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.468684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.468649 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:06.778028 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:06.778002 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm"] Oct 30 18:14:06.779634 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:14:06.779599 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded2e6b15_13d5_4fa8_909b_42ecf5f35c1f.slice/crio-22ed88480cd30e03c95909459de5163ff3e197c183408bdbde03319654c51781 WatchSource:0}: Error finding container 22ed88480cd30e03c95909459de5163ff3e197c183408bdbde03319654c51781: Status 404 returned error can't find the container with id 22ed88480cd30e03c95909459de5163ff3e197c183408bdbde03319654c51781 Oct 30 18:14:07.324806 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:07.324771 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" event={"ID":"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f","Type":"ContainerStarted","Data":"f13b7d4f36eea302662aaf102649aa9b6ac553af0fd2de194b7c46a770302a18"} Oct 30 18:14:07.325236 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:07.324815 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" event={"ID":"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f","Type":"ContainerStarted","Data":"22ed88480cd30e03c95909459de5163ff3e197c183408bdbde03319654c51781"} Oct 30 18:14:07.367544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:07.367472 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podStartSLOduration=1.367457967 podStartE2EDuration="1.367457967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:07.363921363 +0000 UTC m=+746.680341684" watchObservedRunningTime="2026-10-30 18:14:07.367457967 +0000 UTC m=+746.683878288" Oct 30 18:14:07.470453 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:07.470423 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:08.417834 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.417803 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn"] Oct 30 18:14:08.421862 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.421839 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn" Oct 30 18:14:08.439660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.439632 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn"] Oct 30 18:14:08.447100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.447075 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:14:08.447219 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.447139 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:14:08.526593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.526551 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/992df831-f517-479f-8a7b-df3dd1cf31eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f4baac0-jnrdn\" (UID: \"992df831-f517-479f-8a7b-df3dd1cf31eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn" Oct 30 18:14:08.526751 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.526701 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/992df831-f517-479f-8a7b-df3dd1cf31eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f4baac0-jnrdn\" (UID: \"992df831-f517-479f-8a7b-df3dd1cf31eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn" Oct 30 18:14:08.526887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.526865 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bwj9q\" (UniqueName: \"kubernetes.io/projected/992df831-f517-479f-8a7b-df3dd1cf31eb-kube-api-access-bwj9q\") pod \"maas-api-revoke-sub-4f4baac0-jnrdn\" (UID: \"992df831-f517-479f-8a7b-df3dd1cf31eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn" Oct 30 18:14:08.601935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.601901 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w"] Oct 30 18:14:08.606239 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.606216 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.628024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.627993 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/992df831-f517-479f-8a7b-df3dd1cf31eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f4baac0-jnrdn\" (UID: \"992df831-f517-479f-8a7b-df3dd1cf31eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn" Oct 30 18:14:08.628217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.628109 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/992df831-f517-479f-8a7b-df3dd1cf31eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f4baac0-jnrdn\" (UID: \"992df831-f517-479f-8a7b-df3dd1cf31eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn" Oct 30 18:14:08.628293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.628238 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bwj9q\" (UniqueName: \"kubernetes.io/projected/992df831-f517-479f-8a7b-df3dd1cf31eb-kube-api-access-bwj9q\") pod \"maas-api-revoke-sub-4f4baac0-jnrdn\" (UID: \"992df831-f517-479f-8a7b-df3dd1cf31eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn" Oct 30 18:14:08.628721 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.628696 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w"] Oct 30 18:14:08.629363 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.629340 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/992df831-f517-479f-8a7b-df3dd1cf31eb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4f4baac0-jnrdn\" (UID: \"992df831-f517-479f-8a7b-df3dd1cf31eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn" Oct 30 18:14:08.631920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.631899 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/992df831-f517-479f-8a7b-df3dd1cf31eb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4f4baac0-jnrdn\" (UID: \"992df831-f517-479f-8a7b-df3dd1cf31eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn" Oct 30 18:14:08.641474 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.641446 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m"] Oct 30 18:14:08.646713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.646692 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.653592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.653524 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bwj9q\" (UniqueName: \"kubernetes.io/projected/992df831-f517-479f-8a7b-df3dd1cf31eb-kube-api-access-bwj9q\") pod \"maas-api-revoke-sub-4f4baac0-jnrdn\" (UID: \"992df831-f517-479f-8a7b-df3dd1cf31eb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn" Oct 30 18:14:08.654951 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.654915 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m"] Oct 30 18:14:08.729453 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.729414 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jnzm7\" (UniqueName: \"kubernetes.io/projected/c7a2a719-e154-4072-93f2-ed366344dbbc-kube-api-access-jnzm7\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.729453 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.729455 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.729691 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.729558 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.729691 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.729606 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc39faea-f977-4ba2-be13-cef341667507-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.729811 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.729694 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc39faea-f977-4ba2-be13-cef341667507-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.729811 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.729732 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.729811 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.729791 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a2a719-e154-4072-93f2-ed366344dbbc-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.730004 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.729892 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.730004 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.729943 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.730169 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.730016 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc39faea-f977-4ba2-be13-cef341667507-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.730169 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.730042 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.730169 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.730061 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.730169 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.730077 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.730169 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.730096 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.730355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.730262 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.730355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.730296 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.730355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.730322 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-65qkg\" (UniqueName: \"kubernetes.io/projected/cc39faea-f977-4ba2-be13-cef341667507-kube-api-access-65qkg\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.730450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.730364 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.736129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.736108 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn" Oct 30 18:14:08.832099 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.832068 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc39faea-f977-4ba2-be13-cef341667507-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.832278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.832116 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.832278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.832148 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a2a719-e154-4072-93f2-ed366344dbbc-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.832278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.832174 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.832640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.832568 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.832771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.832631 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.832771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.832685 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.832771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.832734 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc39faea-f977-4ba2-be13-cef341667507-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.832894 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.832770 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.832894 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.832796 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.832894 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.832820 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.832894 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.832844 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.833019 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.832934 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a2a719-e154-4072-93f2-ed366344dbbc-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.833019 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.833002 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.833083 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.833030 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.833083 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.833067 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-65qkg\" (UniqueName: \"kubernetes.io/projected/cc39faea-f977-4ba2-be13-cef341667507-kube-api-access-65qkg\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.833144 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.833100 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.833180 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.833134 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-credential-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.833180 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.833152 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jnzm7\" (UniqueName: \"kubernetes.io/projected/c7a2a719-e154-4072-93f2-ed366344dbbc-kube-api-access-jnzm7\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.833242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.833193 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.833242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.833203 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.833305 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.833239 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.833305 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.833295 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc39faea-f977-4ba2-be13-cef341667507-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.833472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.833452 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.833624 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.833604 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-workload-socket\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.833702 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.833685 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc39faea-f977-4ba2-be13-cef341667507-istiod-ca-cert\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.833880 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.833855 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-data\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.834004 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.833918 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-workload-certs\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.836721 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.836697 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.836836 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.836817 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc39faea-f977-4ba2-be13-cef341667507-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.837022 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.836998 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-podinfo\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.840114 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.840089 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-istio-envoy\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.854938 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.854913 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.855745 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.855290 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc39faea-f977-4ba2-be13-cef341667507-istio-token\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.855745 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.855715 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-65qkg\" (UniqueName: \"kubernetes.io/projected/cc39faea-f977-4ba2-be13-cef341667507-kube-api-access-65qkg\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:08.856191 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.856172 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jnzm7\" (UniqueName: \"kubernetes.io/projected/c7a2a719-e154-4072-93f2-ed366344dbbc-kube-api-access-jnzm7\") pod \"e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.921132 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.921053 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:08.977789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:08.977759 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:09.050619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:09.047945 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn"] Oct 30 18:14:09.242745 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:09.242714 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w"] Oct 30 18:14:09.295881 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:09.295858 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m"] Oct 30 18:14:09.311765 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:14:09.311735 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7a2a719_e154_4072_93f2_ed366344dbbc.slice/crio-2bf9ac057f40dec9a91a203c481943ee2e3ef24bf8c41c25e90721c59f6da833 WatchSource:0}: Error finding container 2bf9ac057f40dec9a91a203c481943ee2e3ef24bf8c41c25e90721c59f6da833: Status 404 returned error can't find the container with id 2bf9ac057f40dec9a91a203c481943ee2e3ef24bf8c41c25e90721c59f6da833 Oct 30 18:14:09.313474 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:14:09.313449 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc39faea_f977_4ba2_be13_cef341667507.slice/crio-8bfc688071418547f9692663c15fc0a6158b7472d07fc7e8ee971a705205ed9f WatchSource:0}: Error finding container 8bfc688071418547f9692663c15fc0a6158b7472d07fc7e8ee971a705205ed9f: Status 404 returned error can't find the container with id 8bfc688071418547f9692663c15fc0a6158b7472d07fc7e8ee971a705205ed9f Oct 30 18:14:09.315001 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:09.314888 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:14:09.315001 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:09.314979 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:14:09.315081 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:09.315025 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:14:09.315779 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:09.315750 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:14:09.315869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:09.315814 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:14:09.315869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:09.315856 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:14:09.338884 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:09.338849 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" event={"ID":"cc39faea-f977-4ba2-be13-cef341667507","Type":"ContainerStarted","Data":"8bfc688071418547f9692663c15fc0a6158b7472d07fc7e8ee971a705205ed9f"} Oct 30 18:14:09.340893 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:09.340868 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn" event={"ID":"992df831-f517-479f-8a7b-df3dd1cf31eb","Type":"ContainerStarted","Data":"d8372a819efa7394c17aa7a9113907387e20c31a66093a98f30bebe2ea22743f"} Oct 30 18:14:09.341000 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:09.340923 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn" event={"ID":"992df831-f517-479f-8a7b-df3dd1cf31eb","Type":"ContainerStarted","Data":"06d82ab9db71158cc1f3fdcd5827a9040880fe783f4f701122e12e7a213ac2f4"} Oct 30 18:14:09.342402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:09.342382 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" event={"ID":"c7a2a719-e154-4072-93f2-ed366344dbbc","Type":"ContainerStarted","Data":"2bf9ac057f40dec9a91a203c481943ee2e3ef24bf8c41c25e90721c59f6da833"} Oct 30 18:14:09.383112 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:09.383055 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn" podStartSLOduration=1.383036326 podStartE2EDuration="1.383036326s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:09.379791239 +0000 UTC m=+748.696211562" watchObservedRunningTime="2026-10-30 18:14:09.383036326 +0000 UTC m=+748.699456647" Oct 30 18:14:10.367944 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:10.367843 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" event={"ID":"cc39faea-f977-4ba2-be13-cef341667507","Type":"ContainerStarted","Data":"d561c4c444e09fb77515150263580a4cca237d0bdc7fb8ee2aad28e974ce4aa8"} Oct 30 18:14:10.370286 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:10.370256 5232 generic.go:356] "Generic (PLEG): container finished" podID="992df831-f517-479f-8a7b-df3dd1cf31eb" containerID="d8372a819efa7394c17aa7a9113907387e20c31a66093a98f30bebe2ea22743f" exitCode=0 Oct 30 18:14:10.370405 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:10.370331 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn" event={"ID":"992df831-f517-479f-8a7b-df3dd1cf31eb","Type":"ContainerDied","Data":"d8372a819efa7394c17aa7a9113907387e20c31a66093a98f30bebe2ea22743f"} Oct 30 18:14:10.372097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:10.372074 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" event={"ID":"c7a2a719-e154-4072-93f2-ed366344dbbc","Type":"ContainerStarted","Data":"4b840d76c940168f16b1d2d52fe061fd0f8eeba0102c0483c1df1ad8c95469d3"} Oct 30 18:14:10.445154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:10.445086 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" podStartSLOduration=2.445062185 podStartE2EDuration="2.445062185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:10.442898093 +0000 UTC m=+749.759318412" watchObservedRunningTime="2026-10-30 18:14:10.445062185 +0000 UTC m=+749.761482508" Oct 30 18:14:10.537532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:10.537448 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" podStartSLOduration=2.53742667 podStartE2EDuration="2.53742667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:10.533851669 +0000 UTC m=+749.850272003" watchObservedRunningTime="2026-10-30 18:14:10.53742667 +0000 UTC m=+749.853846993" Oct 30 18:14:10.929535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:10.928277 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:10.939278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:10.938655 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:14:10.939278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:10.938749 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" podUID="c7a2a719-e154-4072-93f2-ed366344dbbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:14:10.978485 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:10.978454 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:10.979559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:10.979533 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:14:10.979683 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:10.979590 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" podUID="cc39faea-f977-4ba2-be13-cef341667507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:14:11.687696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:11.687671 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn" Oct 30 18:14:11.799777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:11.799740 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/992df831-f517-479f-8a7b-df3dd1cf31eb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/992df831-f517-479f-8a7b-df3dd1cf31eb-maas-api-openshift-ca\") pod \"992df831-f517-479f-8a7b-df3dd1cf31eb\" (UID: \"992df831-f517-479f-8a7b-df3dd1cf31eb\") " Oct 30 18:14:11.799946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:11.799826 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/992df831-f517-479f-8a7b-df3dd1cf31eb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/992df831-f517-479f-8a7b-df3dd1cf31eb-maas-api-xks-ca\") pod \"992df831-f517-479f-8a7b-df3dd1cf31eb\" (UID: \"992df831-f517-479f-8a7b-df3dd1cf31eb\") " Oct 30 18:14:11.799946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:11.799852 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992df831-f517-479f-8a7b-df3dd1cf31eb-kube-api-access-bwj9q\" (UniqueName: \"kubernetes.io/projected/992df831-f517-479f-8a7b-df3dd1cf31eb-kube-api-access-bwj9q\") pod \"992df831-f517-479f-8a7b-df3dd1cf31eb\" (UID: \"992df831-f517-479f-8a7b-df3dd1cf31eb\") " Oct 30 18:14:11.800090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:11.800066 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/992df831-f517-479f-8a7b-df3dd1cf31eb-maas-api-openshift-ca" pod "992df831-f517-479f-8a7b-df3dd1cf31eb" (UID: "992df831-f517-479f-8a7b-df3dd1cf31eb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:11.802276 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:11.802249 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992df831-f517-479f-8a7b-df3dd1cf31eb-kube-api-access-bwj9q" pod "992df831-f517-479f-8a7b-df3dd1cf31eb" (UID: "992df831-f517-479f-8a7b-df3dd1cf31eb"). InnerVolumeSpecName "kube-api-access-bwj9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:11.802392 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:11.802330 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/992df831-f517-479f-8a7b-df3dd1cf31eb-maas-api-xks-ca" pod "992df831-f517-479f-8a7b-df3dd1cf31eb" (UID: "992df831-f517-479f-8a7b-df3dd1cf31eb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:11.901103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:11.901068 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/992df831-f517-479f-8a7b-df3dd1cf31eb-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:14:11.901320 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:11.901300 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/992df831-f517-479f-8a7b-df3dd1cf31eb-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:14:11.901387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:11.901325 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bwj9q\" (UniqueName: \"kubernetes.io/projected/992df831-f517-479f-8a7b-df3dd1cf31eb-kube-api-access-bwj9q\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:14:11.923594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:11.923107 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:14:11.923771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:11.923629 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" podUID="c7a2a719-e154-4072-93f2-ed366344dbbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:14:11.978837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:11.978788 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:14:11.979041 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:11.978855 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" podUID="cc39faea-f977-4ba2-be13-cef341667507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:14:12.394008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:12.393968 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn" event={"ID":"992df831-f517-479f-8a7b-df3dd1cf31eb","Type":"ContainerDied","Data":"06d82ab9db71158cc1f3fdcd5827a9040880fe783f4f701122e12e7a213ac2f4"} Oct 30 18:14:12.394200 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:12.394031 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn" Oct 30 18:14:12.394200 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:12.394109 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="06d82ab9db71158cc1f3fdcd5827a9040880fe783f4f701122e12e7a213ac2f4" Oct 30 18:14:12.922909 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:12.922879 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:14:12.923466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:12.923434 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" podUID="c7a2a719-e154-4072-93f2-ed366344dbbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:14:12.978416 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:12.978386 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:14:12.978664 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:12.978645 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" podUID="cc39faea-f977-4ba2-be13-cef341667507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:14:13.164534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.163370 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x"] Oct 30 18:14:13.164534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.164206 5232 cpu_manager.go:414] "containerMap: removing container" podUID="992df831-f517-479f-8a7b-df3dd1cf31eb" containerName="revoke-keys" Oct 30 18:14:13.164534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.164226 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992df831-f517-479f-8a7b-df3dd1cf31eb" containerName="revoke-keys" Oct 30 18:14:13.164534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.164410 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="992df831-f517-479f-8a7b-df3dd1cf31eb" containerName="revoke-keys" Oct 30 18:14:13.266769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.266732 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x"] Oct 30 18:14:13.267104 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.267083 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x" Oct 30 18:14:13.282798 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.282681 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:14:13.282798 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.282717 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:14:13.377797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.377763 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f84cf0c4-gkt8x\" (UID: \"0f5333ad-eb5b-42e0-b748-b1b48a4c00ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x" Oct 30 18:14:13.378007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.377837 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8jzl6\" (UniqueName: \"kubernetes.io/projected/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-kube-api-access-8jzl6\") pod \"maas-api-revoke-sub-f84cf0c4-gkt8x\" (UID: \"0f5333ad-eb5b-42e0-b748-b1b48a4c00ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x" Oct 30 18:14:13.378007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.377903 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f84cf0c4-gkt8x\" (UID: \"0f5333ad-eb5b-42e0-b748-b1b48a4c00ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x" Oct 30 18:14:13.480040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.480008 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8jzl6\" (UniqueName: \"kubernetes.io/projected/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-kube-api-access-8jzl6\") pod \"maas-api-revoke-sub-f84cf0c4-gkt8x\" (UID: \"0f5333ad-eb5b-42e0-b748-b1b48a4c00ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x" Oct 30 18:14:13.480250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.480093 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f84cf0c4-gkt8x\" (UID: \"0f5333ad-eb5b-42e0-b748-b1b48a4c00ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x" Oct 30 18:14:13.480315 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.480274 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f84cf0c4-gkt8x\" (UID: \"0f5333ad-eb5b-42e0-b748-b1b48a4c00ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x" Oct 30 18:14:13.481034 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.481009 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f84cf0c4-gkt8x\" (UID: \"0f5333ad-eb5b-42e0-b748-b1b48a4c00ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x" Oct 30 18:14:13.484428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.484405 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f84cf0c4-gkt8x\" (UID: \"0f5333ad-eb5b-42e0-b748-b1b48a4c00ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x" Oct 30 18:14:13.499300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.499272 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8jzl6\" (UniqueName: \"kubernetes.io/projected/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-kube-api-access-8jzl6\") pod \"maas-api-revoke-sub-f84cf0c4-gkt8x\" (UID: \"0f5333ad-eb5b-42e0-b748-b1b48a4c00ad\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x" Oct 30 18:14:13.589083 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.588988 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x" Oct 30 18:14:13.965562 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.965532 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x"] Oct 30 18:14:13.966023 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.965993 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:14:13.966560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.966085 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" podUID="c7a2a719-e154-4072-93f2-ed366344dbbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:14:13.978471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.978447 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:14:13.978639 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:13.978486 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" podUID="cc39faea-f977-4ba2-be13-cef341667507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:14:14.407294 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:14.407262 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x" event={"ID":"0f5333ad-eb5b-42e0-b748-b1b48a4c00ad","Type":"ContainerStarted","Data":"0eff807152adc1e541153812e2b8a12bd10ec5a8cf943df16acc4983dcd3ac70"} Oct 30 18:14:14.925538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:14.925045 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:14:14.925538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:14.925110 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" podUID="c7a2a719-e154-4072-93f2-ed366344dbbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:14:14.994534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:14.994042 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:14:14.994534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:14.994105 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" podUID="cc39faea-f977-4ba2-be13-cef341667507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:14:15.459211 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:15.453684 5232 generic.go:356] "Generic (PLEG): container finished" podID="0f5333ad-eb5b-42e0-b748-b1b48a4c00ad" containerID="7b9c98fd56714eee92b29b449377d34a0959a1cf6b2d8e731dd4397068863c92" exitCode=0 Oct 30 18:14:15.459671 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:15.458565 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x" event={"ID":"0f5333ad-eb5b-42e0-b748-b1b48a4c00ad","Type":"ContainerDied","Data":"7b9c98fd56714eee92b29b449377d34a0959a1cf6b2d8e731dd4397068863c92"} Oct 30 18:14:15.921680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:15.921644 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" start-of-body= Oct 30 18:14:15.921841 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:15.921698 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" podUID="c7a2a719-e154-4072-93f2-ed366344dbbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": dial tcp 10.133.0.76:15021: connect: connection refused" Oct 30 18:14:15.978701 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:15.978666 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" start-of-body= Oct 30 18:14:15.978878 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:15.978732 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" podUID="cc39faea-f977-4ba2-be13-cef341667507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": dial tcp 10.133.0.77:15021: connect: connection refused" Oct 30 18:14:16.471947 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:16.471913 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:16.537624 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:16.537160 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:16.694335 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:16.694310 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x" Oct 30 18:14:16.752376 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:16.746624 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-maas-api-xks-ca\") pod \"0f5333ad-eb5b-42e0-b748-b1b48a4c00ad\" (UID: \"0f5333ad-eb5b-42e0-b748-b1b48a4c00ad\") " Oct 30 18:14:16.752376 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:16.746860 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-kube-api-access-8jzl6\" (UniqueName: \"kubernetes.io/projected/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-kube-api-access-8jzl6\") pod \"0f5333ad-eb5b-42e0-b748-b1b48a4c00ad\" (UID: \"0f5333ad-eb5b-42e0-b748-b1b48a4c00ad\") " Oct 30 18:14:16.752376 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:16.746911 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-maas-api-openshift-ca\") pod \"0f5333ad-eb5b-42e0-b748-b1b48a4c00ad\" (UID: \"0f5333ad-eb5b-42e0-b748-b1b48a4c00ad\") " Oct 30 18:14:16.767589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:16.762745 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-maas-api-openshift-ca" pod "0f5333ad-eb5b-42e0-b748-b1b48a4c00ad" (UID: "0f5333ad-eb5b-42e0-b748-b1b48a4c00ad"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:16.783704 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:16.783457 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-maas-api-xks-ca" pod "0f5333ad-eb5b-42e0-b748-b1b48a4c00ad" (UID: "0f5333ad-eb5b-42e0-b748-b1b48a4c00ad"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:16.784276 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:16.784246 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-kube-api-access-8jzl6" pod "0f5333ad-eb5b-42e0-b748-b1b48a4c00ad" (UID: "0f5333ad-eb5b-42e0-b748-b1b48a4c00ad"). InnerVolumeSpecName "kube-api-access-8jzl6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:16.848296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:16.848261 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:14:16.848296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:16.848294 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8jzl6\" (UniqueName: \"kubernetes.io/projected/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-kube-api-access-8jzl6\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:14:16.848296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:16.848305 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:14:16.926735 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:16.926702 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:16.927196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:16.927170 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:16.927975 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:16.927958 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:14:16.983587 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:16.983559 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:16.983845 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:16.983826 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:16.984537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:16.984493 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:14:17.487089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:17.487060 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:17.547079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:17.547042 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x" Oct 30 18:14:17.547700 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:17.547676 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x" event={"ID":"0f5333ad-eb5b-42e0-b748-b1b48a4c00ad","Type":"ContainerDied","Data":"0eff807152adc1e541153812e2b8a12bd10ec5a8cf943df16acc4983dcd3ac70"} Oct 30 18:14:17.547792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:17.547724 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0eff807152adc1e541153812e2b8a12bd10ec5a8cf943df16acc4983dcd3ac70" Oct 30 18:14:23.193664 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.193630 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp"] Oct 30 18:14:23.194392 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.194366 5232 cpu_manager.go:414] "containerMap: removing container" podUID="0f5333ad-eb5b-42e0-b748-b1b48a4c00ad" containerName="revoke-keys" Oct 30 18:14:23.194558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.194404 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f5333ad-eb5b-42e0-b748-b1b48a4c00ad" containerName="revoke-keys" Oct 30 18:14:23.194629 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.194582 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f5333ad-eb5b-42e0-b748-b1b48a4c00ad" containerName="revoke-keys" Oct 30 18:14:23.199561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.199542 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp" Oct 30 18:14:23.214464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.214436 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp"] Oct 30 18:14:23.222056 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.221874 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:14:23.222056 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.221895 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:14:23.238895 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.238862 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fd2aa8-49b1-432b-9adf-2381ff3451b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8c58813-569dp\" (UID: \"94fd2aa8-49b1-432b-9adf-2381ff3451b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp" Oct 30 18:14:23.239034 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.238928 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fd2aa8-49b1-432b-9adf-2381ff3451b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8c58813-569dp\" (UID: \"94fd2aa8-49b1-432b-9adf-2381ff3451b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp" Oct 30 18:14:23.239080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.239037 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kf6sg\" (UniqueName: \"kubernetes.io/projected/94fd2aa8-49b1-432b-9adf-2381ff3451b0-kube-api-access-kf6sg\") pod \"maas-api-revoke-sub-d8c58813-569dp\" (UID: \"94fd2aa8-49b1-432b-9adf-2381ff3451b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp" Oct 30 18:14:23.340314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.340281 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fd2aa8-49b1-432b-9adf-2381ff3451b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8c58813-569dp\" (UID: \"94fd2aa8-49b1-432b-9adf-2381ff3451b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp" Oct 30 18:14:23.340484 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.340327 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fd2aa8-49b1-432b-9adf-2381ff3451b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8c58813-569dp\" (UID: \"94fd2aa8-49b1-432b-9adf-2381ff3451b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp" Oct 30 18:14:23.340484 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.340355 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kf6sg\" (UniqueName: \"kubernetes.io/projected/94fd2aa8-49b1-432b-9adf-2381ff3451b0-kube-api-access-kf6sg\") pod \"maas-api-revoke-sub-d8c58813-569dp\" (UID: \"94fd2aa8-49b1-432b-9adf-2381ff3451b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp" Oct 30 18:14:23.340953 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.340930 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fd2aa8-49b1-432b-9adf-2381ff3451b0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d8c58813-569dp\" (UID: \"94fd2aa8-49b1-432b-9adf-2381ff3451b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp" Oct 30 18:14:23.342840 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.342821 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fd2aa8-49b1-432b-9adf-2381ff3451b0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d8c58813-569dp\" (UID: \"94fd2aa8-49b1-432b-9adf-2381ff3451b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp" Oct 30 18:14:23.359366 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.359335 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kf6sg\" (UniqueName: \"kubernetes.io/projected/94fd2aa8-49b1-432b-9adf-2381ff3451b0-kube-api-access-kf6sg\") pod \"maas-api-revoke-sub-d8c58813-569dp\" (UID: \"94fd2aa8-49b1-432b-9adf-2381ff3451b0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp" Oct 30 18:14:23.513884 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.513851 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp" Oct 30 18:14:23.803007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.802939 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4"] Oct 30 18:14:23.807560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.807496 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.856656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.842834 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4"] Oct 30 18:14:23.856656 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:14:23.843367 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94fd2aa8_49b1_432b_9adf_2381ff3451b0.slice/crio-28a0601b78a06f3641b8f425b8f97cf09ed337e1284c85e5ce4fb052f1a718b4 WatchSource:0}: Error finding container 28a0601b78a06f3641b8f425b8f97cf09ed337e1284c85e5ce4fb052f1a718b4: Status 404 returned error can't find the container with id 28a0601b78a06f3641b8f425b8f97cf09ed337e1284c85e5ce4fb052f1a718b4 Oct 30 18:14:23.856656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.850916 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp"] Oct 30 18:14:23.856656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.854101 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.856656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.854133 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.856656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.854220 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae13a01-7f20-4354-8771-6d65d6695651-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.856656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.854256 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.856656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.854314 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mvdpm\" (UniqueName: \"kubernetes.io/projected/bae13a01-7f20-4354-8771-6d65d6695651-kube-api-access-mvdpm\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.856656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.854339 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae13a01-7f20-4354-8771-6d65d6695651-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.856656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.854355 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.856656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.854409 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.856656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.854425 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae13a01-7f20-4354-8771-6d65d6695651-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.955336 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.955303 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mvdpm\" (UniqueName: \"kubernetes.io/projected/bae13a01-7f20-4354-8771-6d65d6695651-kube-api-access-mvdpm\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.955479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.955348 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae13a01-7f20-4354-8771-6d65d6695651-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.955606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.955465 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.955655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.955609 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.955691 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.955641 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae13a01-7f20-4354-8771-6d65d6695651-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.955772 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.955752 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.955828 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.955791 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.955918 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.955871 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.956051 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.955971 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae13a01-7f20-4354-8771-6d65d6695651-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.956120 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.956045 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae13a01-7f20-4354-8771-6d65d6695651-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.956120 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.956065 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.956120 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.956092 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.956278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.956158 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.956376 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.956358 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.958270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.958245 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.958633 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.958614 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae13a01-7f20-4354-8771-6d65d6695651-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.974628 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.974607 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae13a01-7f20-4354-8771-6d65d6695651-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:23.974725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:23.974676 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mvdpm\" (UniqueName: \"kubernetes.io/projected/bae13a01-7f20-4354-8771-6d65d6695651-kube-api-access-mvdpm\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:24.139545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:24.139484 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:24.449798 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:24.449771 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4"] Oct 30 18:14:24.452130 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:14:24.452102 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbae13a01_7f20_4354_8771_6d65d6695651.slice/crio-9de8b21475441cf3c4816f1c7cbb4170a0a4e30fef183525c258d66ab7b0c2ca WatchSource:0}: Error finding container 9de8b21475441cf3c4816f1c7cbb4170a0a4e30fef183525c258d66ab7b0c2ca: Status 404 returned error can't find the container with id 9de8b21475441cf3c4816f1c7cbb4170a0a4e30fef183525c258d66ab7b0c2ca Oct 30 18:14:24.454797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:24.454765 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:14:24.454906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:24.454831 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:14:24.454906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:24.454863 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:14:24.653030 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:24.649034 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" event={"ID":"bae13a01-7f20-4354-8771-6d65d6695651","Type":"ContainerStarted","Data":"4affc61103204172c0d502385caa862c21fdeeb0804a5953e7dadf8ebea37875"} Oct 30 18:14:24.653030 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:24.649105 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" event={"ID":"bae13a01-7f20-4354-8771-6d65d6695651","Type":"ContainerStarted","Data":"9de8b21475441cf3c4816f1c7cbb4170a0a4e30fef183525c258d66ab7b0c2ca"} Oct 30 18:14:24.672568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:24.671948 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp" event={"ID":"94fd2aa8-49b1-432b-9adf-2381ff3451b0","Type":"ContainerStarted","Data":"f16ffafe13049216251fa2556cb8160315f4bc95c6ceb91f7408047f91806ccc"} Oct 30 18:14:24.672568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:24.672015 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp" event={"ID":"94fd2aa8-49b1-432b-9adf-2381ff3451b0","Type":"ContainerStarted","Data":"28a0601b78a06f3641b8f425b8f97cf09ed337e1284c85e5ce4fb052f1a718b4"} Oct 30 18:14:24.739282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:24.739221 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" podStartSLOduration=1.73920224 podStartE2EDuration="1.73920224s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:24.701796616 +0000 UTC m=+764.018216949" watchObservedRunningTime="2026-10-30 18:14:24.73920224 +0000 UTC m=+764.055622561" Oct 30 18:14:24.743461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:24.743411 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp" podStartSLOduration=1.743390387 podStartE2EDuration="1.743390387s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:24.738759183 +0000 UTC m=+764.055179504" watchObservedRunningTime="2026-10-30 18:14:24.743390387 +0000 UTC m=+764.059810709" Oct 30 18:14:25.140609 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:25.140568 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:25.142443 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:25.142415 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:25.142606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:25.142463 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" podUID="bae13a01-7f20-4354-8771-6d65d6695651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:14:25.680477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:25.680436 5232 generic.go:356] "Generic (PLEG): container finished" podID="94fd2aa8-49b1-432b-9adf-2381ff3451b0" containerID="f16ffafe13049216251fa2556cb8160315f4bc95c6ceb91f7408047f91806ccc" exitCode=0 Oct 30 18:14:25.680961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:25.680539 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp" event={"ID":"94fd2aa8-49b1-432b-9adf-2381ff3451b0","Type":"ContainerDied","Data":"f16ffafe13049216251fa2556cb8160315f4bc95c6ceb91f7408047f91806ccc"} Oct 30 18:14:26.140776 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:26.140737 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:26.140930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:26.140796 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" podUID="bae13a01-7f20-4354-8771-6d65d6695651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:14:26.872190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:26.872167 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp" Oct 30 18:14:26.899154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:26.899127 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94fd2aa8-49b1-432b-9adf-2381ff3451b0-kube-api-access-kf6sg\" (UniqueName: \"kubernetes.io/projected/94fd2aa8-49b1-432b-9adf-2381ff3451b0-kube-api-access-kf6sg\") pod \"94fd2aa8-49b1-432b-9adf-2381ff3451b0\" (UID: \"94fd2aa8-49b1-432b-9adf-2381ff3451b0\") " Oct 30 18:14:26.899280 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:26.899217 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94fd2aa8-49b1-432b-9adf-2381ff3451b0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fd2aa8-49b1-432b-9adf-2381ff3451b0-maas-api-openshift-ca\") pod \"94fd2aa8-49b1-432b-9adf-2381ff3451b0\" (UID: \"94fd2aa8-49b1-432b-9adf-2381ff3451b0\") " Oct 30 18:14:26.899280 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:26.899272 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/94fd2aa8-49b1-432b-9adf-2381ff3451b0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fd2aa8-49b1-432b-9adf-2381ff3451b0-maas-api-xks-ca\") pod \"94fd2aa8-49b1-432b-9adf-2381ff3451b0\" (UID: \"94fd2aa8-49b1-432b-9adf-2381ff3451b0\") " Oct 30 18:14:26.899602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:26.899572 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94fd2aa8-49b1-432b-9adf-2381ff3451b0-maas-api-openshift-ca" pod "94fd2aa8-49b1-432b-9adf-2381ff3451b0" (UID: "94fd2aa8-49b1-432b-9adf-2381ff3451b0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:14:26.900071 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:26.899882 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/94fd2aa8-49b1-432b-9adf-2381ff3451b0-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:14:26.901584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:26.901563 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/94fd2aa8-49b1-432b-9adf-2381ff3451b0-maas-api-xks-ca" pod "94fd2aa8-49b1-432b-9adf-2381ff3451b0" (UID: "94fd2aa8-49b1-432b-9adf-2381ff3451b0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:14:26.901644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:26.901588 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94fd2aa8-49b1-432b-9adf-2381ff3451b0-kube-api-access-kf6sg" pod "94fd2aa8-49b1-432b-9adf-2381ff3451b0" (UID: "94fd2aa8-49b1-432b-9adf-2381ff3451b0"). InnerVolumeSpecName "kube-api-access-kf6sg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:14:27.001401 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:27.001374 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/94fd2aa8-49b1-432b-9adf-2381ff3451b0-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:14:27.001401 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:27.001402 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kf6sg\" (UniqueName: \"kubernetes.io/projected/94fd2aa8-49b1-432b-9adf-2381ff3451b0-kube-api-access-kf6sg\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:14:27.049278 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:14:27.049242 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:14:27.140411 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:27.140374 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:27.140613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:27.140447 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" podUID="bae13a01-7f20-4354-8771-6d65d6695651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:14:27.694732 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:27.694694 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp" event={"ID":"94fd2aa8-49b1-432b-9adf-2381ff3451b0","Type":"ContainerDied","Data":"28a0601b78a06f3641b8f425b8f97cf09ed337e1284c85e5ce4fb052f1a718b4"} Oct 30 18:14:27.694732 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:27.694713 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp" Oct 30 18:14:27.694948 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:27.694744 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="28a0601b78a06f3641b8f425b8f97cf09ed337e1284c85e5ce4fb052f1a718b4" Oct 30 18:14:28.140674 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.140640 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:28.141050 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.140726 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" podUID="bae13a01-7f20-4354-8771-6d65d6695651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:14:28.738178 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.738141 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz"] Oct 30 18:14:28.738888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.738870 5232 cpu_manager.go:414] "containerMap: removing container" podUID="94fd2aa8-49b1-432b-9adf-2381ff3451b0" containerName="revoke-keys" Oct 30 18:14:28.738954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.738891 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94fd2aa8-49b1-432b-9adf-2381ff3451b0" containerName="revoke-keys" Oct 30 18:14:28.738997 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.738988 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="94fd2aa8-49b1-432b-9adf-2381ff3451b0" containerName="revoke-keys" Oct 30 18:14:28.742850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.742827 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:28.759631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.759600 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz"] Oct 30 18:14:28.765093 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.765075 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8b6cf1\"/\"distinct-w3-8b6cf1-kserve-self-signed-certs\"" Oct 30 18:14:28.856228 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.856193 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-dshm\") pod \"distinct-w3-8b6cf1-kserve-58946f4c46-ksftz\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:28.856405 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.856321 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhmsq\" (UniqueName: \"kubernetes.io/projected/1ade2fbb-e0fe-4957-a7ff-a74990256474-kube-api-access-xhmsq\") pod \"distinct-w3-8b6cf1-kserve-58946f4c46-ksftz\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:28.856454 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.856426 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-home\") pod \"distinct-w3-8b6cf1-kserve-58946f4c46-ksftz\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:28.856542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.856520 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ade2fbb-e0fe-4957-a7ff-a74990256474-tls-certs\") pod \"distinct-w3-8b6cf1-kserve-58946f4c46-ksftz\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:28.856643 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.856626 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-tmp-dir\") pod \"distinct-w3-8b6cf1-kserve-58946f4c46-ksftz\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:28.856711 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.856675 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-model-cache\") pod \"distinct-w3-8b6cf1-kserve-58946f4c46-ksftz\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:28.958107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.958073 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-dshm\") pod \"distinct-w3-8b6cf1-kserve-58946f4c46-ksftz\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:28.958279 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.958192 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhmsq\" (UniqueName: \"kubernetes.io/projected/1ade2fbb-e0fe-4957-a7ff-a74990256474-kube-api-access-xhmsq\") pod \"distinct-w3-8b6cf1-kserve-58946f4c46-ksftz\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:28.958365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.958345 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-home\") pod \"distinct-w3-8b6cf1-kserve-58946f4c46-ksftz\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:28.958461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.958398 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ade2fbb-e0fe-4957-a7ff-a74990256474-tls-certs\") pod \"distinct-w3-8b6cf1-kserve-58946f4c46-ksftz\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:28.958555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.958479 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-tmp-dir\") pod \"distinct-w3-8b6cf1-kserve-58946f4c46-ksftz\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:28.958555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.958533 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-model-cache\") pod \"distinct-w3-8b6cf1-kserve-58946f4c46-ksftz\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:28.958840 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.958807 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-home\") pod \"distinct-w3-8b6cf1-kserve-58946f4c46-ksftz\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:28.958958 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.958820 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-tmp-dir\") pod \"distinct-w3-8b6cf1-kserve-58946f4c46-ksftz\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:28.958958 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.958861 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-model-cache\") pod \"distinct-w3-8b6cf1-kserve-58946f4c46-ksftz\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:28.960575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.960552 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-dshm\") pod \"distinct-w3-8b6cf1-kserve-58946f4c46-ksftz\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:28.961123 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.961102 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ade2fbb-e0fe-4957-a7ff-a74990256474-tls-certs\") pod \"distinct-w3-8b6cf1-kserve-58946f4c46-ksftz\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:28.978590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:28.978559 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhmsq\" (UniqueName: \"kubernetes.io/projected/1ade2fbb-e0fe-4957-a7ff-a74990256474-kube-api-access-xhmsq\") pod \"distinct-w3-8b6cf1-kserve-58946f4c46-ksftz\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:29.058816 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:29.058726 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:29.141237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:29.141176 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:29.141946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:29.141822 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" podUID="bae13a01-7f20-4354-8771-6d65d6695651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:14:29.945558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:29.935897 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz"] Oct 30 18:14:30.156031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:30.155946 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:30.156622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:30.156587 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" podUID="bae13a01-7f20-4354-8771-6d65d6695651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:14:30.368661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:30.368609 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 18:14:30.572537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:30.563558 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:14:30.892522 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:30.892398 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:30.997531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:30.991254 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" event={"ID":"1ade2fbb-e0fe-4957-a7ff-a74990256474","Type":"ContainerStarted","Data":"9238a95bdefc3bce3d3352a10b8237c1450926fdb4853ee0d06307d886484c97"} Oct 30 18:14:30.997531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:30.997325 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:31.154182 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:31.154087 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:31.154479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:31.154429 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" podUID="bae13a01-7f20-4354-8771-6d65d6695651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:14:31.196643 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:31.196591 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:31.373532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:31.370076 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 18:14:31.373532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:31.370448 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:14:31.523663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:31.523612 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:31.593087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:31.589827 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:31.593087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:31.590265 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:14:31.718615 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:31.480341 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:31.948037 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:31.947835 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:32.106027 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:32.105994 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:14:32.190540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:32.165499 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:32.190885 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:32.190846 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" podUID="bae13a01-7f20-4354-8771-6d65d6695651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:14:32.279310 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:32.278396 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:14:32.279310 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:32.278991 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:14:32.375934 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:32.356696 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:14:32.375934 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:32.375754 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" podUID="5d5ba31c-c8f3-4467-be05-fd146702503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:32.375934 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:32.375866 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:32.536601 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:32.535403 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:32.557566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:32.537310 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:14:32.603460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:32.596660 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:32.722735 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:32.722617 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:14:32.727535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:32.723716 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:33.062875 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:33.055612 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:14:33.142175 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:33.142153 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:14:33.143973 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:33.143950 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" start-of-body= Oct 30 18:14:33.144210 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:33.144182 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" podUID="bae13a01-7f20-4354-8771-6d65d6695651" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.80:15021/healthz/ready\": dial tcp 10.133.0.80:15021: connect: connection refused" Oct 30 18:14:33.146847 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:33.146829 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:14:33.194534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:33.174044 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:14:33.207759 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:33.207724 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:14:33.209406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:33.209373 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:14:34.249751 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:34.249715 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" event={"ID":"1ade2fbb-e0fe-4957-a7ff-a74990256474","Type":"ContainerStarted","Data":"b0332b10b4ab6eb47cd6dccae366dc9a91d7bb2c384857f58eae8214ca10b7a0"} Oct 30 18:14:34.259558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:34.257713 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:34.259558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:34.258938 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:34.260212 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:34.260192 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:14:34.289501 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:34.289440 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podStartSLOduration=6.289422589 podStartE2EDuration="6.289422589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:34.287117211 +0000 UTC m=+773.603537533" watchObservedRunningTime="2026-10-30 18:14:34.289422589 +0000 UTC m=+773.605842917" Oct 30 18:14:35.059423 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:35.059387 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:38.787987 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.787956 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6"] Oct 30 18:14:38.792688 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.792665 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.816125 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.815870 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6"] Oct 30 18:14:38.834260 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.834223 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tr8sh\" (UniqueName: \"kubernetes.io/projected/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-kube-api-access-tr8sh\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.834410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.834357 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.834410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.834398 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.834559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.834453 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.834621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.834556 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.834743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.834715 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.835218 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.834783 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.835218 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.834836 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.835218 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.834872 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.949817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.949779 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.957290 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.957263 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.958001 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.957974 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.958490 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.957144 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.959101 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.958409 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.959281 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.958704 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tr8sh\" (UniqueName: \"kubernetes.io/projected/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-kube-api-access-tr8sh\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.959480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.959462 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.959937 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.959921 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.960781 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.960762 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.960916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.960899 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.961790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.961769 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.962310 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.959855 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.962446 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.960699 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.962568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.961233 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.987337 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.987303 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.988928 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.988896 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.991825 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.991789 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tr8sh\" (UniqueName: \"kubernetes.io/projected/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-kube-api-access-tr8sh\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:38.992101 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:38.992083 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:39.059362 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:39.059283 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:39.072445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:39.072419 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:39.106433 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:39.106404 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:39.294928 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:39.294896 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:14:39.430490 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:39.430466 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6"] Oct 30 18:14:39.433180 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:14:39.433150 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0bbd2e9b_4989_46ca_bf9b_afe2ef671726.slice/crio-23a2c4e81093764cc3785180178c87a3cc9c883ad37ac8358fe4b1091d7aef1a WatchSource:0}: Error finding container 23a2c4e81093764cc3785180178c87a3cc9c883ad37ac8358fe4b1091d7aef1a: Status 404 returned error can't find the container with id 23a2c4e81093764cc3785180178c87a3cc9c883ad37ac8358fe4b1091d7aef1a Oct 30 18:14:39.435927 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:39.435896 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:14:39.435995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:39.435966 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:14:39.436033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:39.436000 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:14:40.292235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:40.292194 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" event={"ID":"0bbd2e9b-4989-46ca-bf9b-afe2ef671726","Type":"ContainerStarted","Data":"660df6d969acdecc8c744f0e7aab5c6755bb3717fede45654cc9261893ebcbf0"} Oct 30 18:14:40.292658 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:40.292259 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" event={"ID":"0bbd2e9b-4989-46ca-bf9b-afe2ef671726","Type":"ContainerStarted","Data":"23a2c4e81093764cc3785180178c87a3cc9c883ad37ac8358fe4b1091d7aef1a"} Oct 30 18:14:40.359743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:40.359684 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" podStartSLOduration=2.359667305 podStartE2EDuration="2.359667305s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:14:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:14:40.348275138 +0000 UTC m=+779.664695482" watchObservedRunningTime="2026-10-30 18:14:40.359667305 +0000 UTC m=+779.676087626" Oct 30 18:14:41.107126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:41.107090 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:41.108303 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:41.108274 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:41.108454 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:41.108335 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" podUID="0bbd2e9b-4989-46ca-bf9b-afe2ef671726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 18:14:42.107102 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:42.107060 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:42.107604 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:42.107129 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" podUID="0bbd2e9b-4989-46ca-bf9b-afe2ef671726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 18:14:43.107012 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:43.106970 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:43.107192 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:43.107038 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" podUID="0bbd2e9b-4989-46ca-bf9b-afe2ef671726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 18:14:44.107749 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:44.107716 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:44.108142 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:44.107771 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" podUID="0bbd2e9b-4989-46ca-bf9b-afe2ef671726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 18:14:45.107451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:45.107415 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:45.107690 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:45.107477 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" podUID="0bbd2e9b-4989-46ca-bf9b-afe2ef671726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 18:14:46.107157 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:46.107118 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" start-of-body= Oct 30 18:14:46.107637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:46.107224 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" podUID="0bbd2e9b-4989-46ca-bf9b-afe2ef671726" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.82:15021/healthz/ready\": dial tcp 10.133.0.82:15021: connect: connection refused" Oct 30 18:14:47.111159 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:47.111133 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:47.111642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:47.111623 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:47.112269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:14:47.112253 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:14:57.033795 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:14:57.033748 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:15:00.188535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.187743 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw"] Oct 30 18:15:00.216839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.216805 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" Oct 30 18:15:00.253547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.237393 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:15:00.512829 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.512779 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7c2d8a-88e9-451c-938a-56a685d17a26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw\" (UID: \"ab7c2d8a-88e9-451c-938a-56a685d17a26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" Oct 30 18:15:00.526076 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.526041 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pftg8\" (UniqueName: \"kubernetes.io/projected/ab7c2d8a-88e9-451c-938a-56a685d17a26-kube-api-access-pftg8\") pod \"maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw\" (UID: \"ab7c2d8a-88e9-451c-938a-56a685d17a26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" Oct 30 18:15:00.528993 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.523678 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6"] Oct 30 18:15:00.584864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.584831 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" Oct 30 18:15:00.595372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.595346 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb"] Oct 30 18:15:00.630783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.630744 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-whr5w\" (UniqueName: \"kubernetes.io/projected/44a3dd61-3a4f-40b9-8c5f-0896c37c5029-kube-api-access-whr5w\") pod \"maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6\" (UID: \"44a3dd61-3a4f-40b9-8c5f-0896c37c5029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" Oct 30 18:15:00.634678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.634648 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pftg8\" (UniqueName: \"kubernetes.io/projected/ab7c2d8a-88e9-451c-938a-56a685d17a26-kube-api-access-pftg8\") pod \"maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw\" (UID: \"ab7c2d8a-88e9-451c-938a-56a685d17a26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" Oct 30 18:15:00.634904 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.634884 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44a3dd61-3a4f-40b9-8c5f-0896c37c5029-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6\" (UID: \"44a3dd61-3a4f-40b9-8c5f-0896c37c5029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" Oct 30 18:15:00.636663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.635113 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7c2d8a-88e9-451c-938a-56a685d17a26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw\" (UID: \"ab7c2d8a-88e9-451c-938a-56a685d17a26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" Oct 30 18:15:00.646693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.637037 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7c2d8a-88e9-451c-938a-56a685d17a26-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw\" (UID: \"ab7c2d8a-88e9-451c-938a-56a685d17a26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" Oct 30 18:15:00.649423 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.649396 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v"] Oct 30 18:15:00.676218 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.649705 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" Oct 30 18:15:00.676547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.676494 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pftg8\" (UniqueName: \"kubernetes.io/projected/ab7c2d8a-88e9-451c-938a-56a685d17a26-kube-api-access-pftg8\") pod \"maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw\" (UID: \"ab7c2d8a-88e9-451c-938a-56a685d17a26\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" Oct 30 18:15:00.700565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.700485 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2"] Oct 30 18:15:00.707801 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.701940 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" Oct 30 18:15:00.767410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.767330 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44a3dd61-3a4f-40b9-8c5f-0896c37c5029-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6\" (UID: \"44a3dd61-3a4f-40b9-8c5f-0896c37c5029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" Oct 30 18:15:00.767722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.767645 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34bb7979-7eb6-4b2f-a7d2-90acc05254f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v\" (UID: \"34bb7979-7eb6-4b2f-a7d2-90acc05254f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" Oct 30 18:15:00.768642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.768613 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44a3dd61-3a4f-40b9-8c5f-0896c37c5029-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6\" (UID: \"44a3dd61-3a4f-40b9-8c5f-0896c37c5029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" Oct 30 18:15:00.768795 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.768760 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czw9z\" (UniqueName: \"kubernetes.io/projected/3060b110-ab37-472b-b82a-c360d5bbd9a8-kube-api-access-czw9z\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb\" (UID: \"3060b110-ab37-472b-b82a-c360d5bbd9a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" Oct 30 18:15:00.769127 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.769107 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3060b110-ab37-472b-b82a-c360d5bbd9a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb\" (UID: \"3060b110-ab37-472b-b82a-c360d5bbd9a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" Oct 30 18:15:00.772531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.769345 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4stz\" (UniqueName: \"kubernetes.io/projected/34bb7979-7eb6-4b2f-a7d2-90acc05254f4-kube-api-access-g4stz\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v\" (UID: \"34bb7979-7eb6-4b2f-a7d2-90acc05254f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" Oct 30 18:15:00.772889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.772869 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-whr5w\" (UniqueName: \"kubernetes.io/projected/44a3dd61-3a4f-40b9-8c5f-0896c37c5029-kube-api-access-whr5w\") pod \"maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6\" (UID: \"44a3dd61-3a4f-40b9-8c5f-0896c37c5029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" Oct 30 18:15:00.776527 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.772210 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" Oct 30 18:15:00.789535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.777312 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6"] Oct 30 18:15:00.840366 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.840330 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-whr5w\" (UniqueName: \"kubernetes.io/projected/44a3dd61-3a4f-40b9-8c5f-0896c37c5029-kube-api-access-whr5w\") pod \"maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6\" (UID: \"44a3dd61-3a4f-40b9-8c5f-0896c37c5029\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" Oct 30 18:15:00.875653 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.875617 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3060b110-ab37-472b-b82a-c360d5bbd9a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb\" (UID: \"3060b110-ab37-472b-b82a-c360d5bbd9a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" Oct 30 18:15:00.912761 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.912730 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4stz\" (UniqueName: \"kubernetes.io/projected/34bb7979-7eb6-4b2f-a7d2-90acc05254f4-kube-api-access-g4stz\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v\" (UID: \"34bb7979-7eb6-4b2f-a7d2-90acc05254f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" Oct 30 18:15:00.930489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.876750 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" Oct 30 18:15:00.930707 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.876604 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3060b110-ab37-472b-b82a-c360d5bbd9a8-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb\" (UID: \"3060b110-ab37-472b-b82a-c360d5bbd9a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" Oct 30 18:15:00.930855 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.930451 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/367e7ada-dcd6-47fc-9117-76a1d38c84ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2\" (UID: \"367e7ada-dcd6-47fc-9117-76a1d38c84ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" Oct 30 18:15:00.931056 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.931031 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5fwz\" (UniqueName: \"kubernetes.io/projected/367e7ada-dcd6-47fc-9117-76a1d38c84ed-kube-api-access-b5fwz\") pod \"maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2\" (UID: \"367e7ada-dcd6-47fc-9117-76a1d38c84ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" Oct 30 18:15:00.931308 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.931286 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34bb7979-7eb6-4b2f-a7d2-90acc05254f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v\" (UID: \"34bb7979-7eb6-4b2f-a7d2-90acc05254f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" Oct 30 18:15:00.932312 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.932292 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czw9z\" (UniqueName: \"kubernetes.io/projected/3060b110-ab37-472b-b82a-c360d5bbd9a8-kube-api-access-czw9z\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb\" (UID: \"3060b110-ab37-472b-b82a-c360d5bbd9a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" Oct 30 18:15:01.031862 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.932196 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34bb7979-7eb6-4b2f-a7d2-90acc05254f4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v\" (UID: \"34bb7979-7eb6-4b2f-a7d2-90acc05254f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" Oct 30 18:15:01.049072 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:00.946797 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw"] Oct 30 18:15:01.049282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:01.032196 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" Oct 30 18:15:01.058539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:01.057229 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czw9z\" (UniqueName: \"kubernetes.io/projected/3060b110-ab37-472b-b82a-c360d5bbd9a8-kube-api-access-czw9z\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb\" (UID: \"3060b110-ab37-472b-b82a-c360d5bbd9a8\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" Oct 30 18:15:01.075547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:01.066086 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4stz\" (UniqueName: \"kubernetes.io/projected/34bb7979-7eb6-4b2f-a7d2-90acc05254f4-kube-api-access-g4stz\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v\" (UID: \"34bb7979-7eb6-4b2f-a7d2-90acc05254f4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" Oct 30 18:15:01.084035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:01.068651 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb"] Oct 30 18:15:01.088790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:01.088756 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" Oct 30 18:15:01.207077 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:01.207043 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" Oct 30 18:15:01.227075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:01.227041 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/367e7ada-dcd6-47fc-9117-76a1d38c84ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2\" (UID: \"367e7ada-dcd6-47fc-9117-76a1d38c84ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" Oct 30 18:15:01.227340 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:01.227317 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5fwz\" (UniqueName: \"kubernetes.io/projected/367e7ada-dcd6-47fc-9117-76a1d38c84ed-kube-api-access-b5fwz\") pod \"maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2\" (UID: \"367e7ada-dcd6-47fc-9117-76a1d38c84ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" Oct 30 18:15:01.232589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:01.232554 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/367e7ada-dcd6-47fc-9117-76a1d38c84ed-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2\" (UID: \"367e7ada-dcd6-47fc-9117-76a1d38c84ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" Oct 30 18:15:01.263550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:01.257713 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5fwz\" (UniqueName: \"kubernetes.io/projected/367e7ada-dcd6-47fc-9117-76a1d38c84ed-kube-api-access-b5fwz\") pod \"maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2\" (UID: \"367e7ada-dcd6-47fc-9117-76a1d38c84ed\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" Oct 30 18:15:01.375050 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:01.374941 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v"] Oct 30 18:15:01.434696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:01.434649 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2"] Oct 30 18:15:01.544923 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:01.544631 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" Oct 30 18:15:02.653822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:02.650626 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:02.653822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:02.650689 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" podUID="f9270d40-a0b1-4c84-a28b-9f4bca0487be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:02.653822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:02.651022 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:02.812020 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:02.809637 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:03.193443 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:03.193364 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:03.199926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:03.193979 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6"] Oct 30 18:15:03.199926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:03.197396 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:03.200717 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:03.200331 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:03.204541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:03.200835 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" podUID="45299d3c-52f3-4d0c-8dd3-af63948cf587" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.66:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:03.443896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:03.443800 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:15:03.474564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:03.474524 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" podUID="a337e101-6a41-4416-9731-f28cf2afd0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:03.704663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:03.703391 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb"] Oct 30 18:15:03.714498 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:03.714465 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw"] Oct 30 18:15:03.757584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:03.757558 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2"] Oct 30 18:15:03.763766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:03.763358 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v"] Oct 30 18:15:03.880759 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:03.880721 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" event={"ID":"44a3dd61-3a4f-40b9-8c5f-0896c37c5029","Type":"ContainerStarted","Data":"4037551a184962e299564d38298a05c030733eeac84338e1407ee187835107b9"} Oct 30 18:15:03.882132 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:03.882096 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" event={"ID":"ab7c2d8a-88e9-451c-938a-56a685d17a26","Type":"ContainerStarted","Data":"d3e2b6a42d47d5151bb1c388068bff2fd2341fb493d91ce1bcf3952e3b8be66d"} Oct 30 18:15:03.883361 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:03.883335 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" event={"ID":"34bb7979-7eb6-4b2f-a7d2-90acc05254f4","Type":"ContainerStarted","Data":"2277aff8c418e160108965a174f6363a3b5e7254dd47671965b1ae59945f3f91"} Oct 30 18:15:03.884563 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:03.884535 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" event={"ID":"367e7ada-dcd6-47fc-9117-76a1d38c84ed","Type":"ContainerStarted","Data":"77d0406acc985f0c434d9dd07199c99f84f08bef1e8df536a0c56bd7b1df56d3"} Oct 30 18:15:03.885964 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:03.885941 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" event={"ID":"3060b110-ab37-472b-b82a-c360d5bbd9a8","Type":"ContainerStarted","Data":"7b4212158b5c06cf7909493c242aeb3bed4fed15c842d0ab4f8d77777f1c3972"} Oct 30 18:15:04.894940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:04.894853 5232 generic.go:356] "Generic (PLEG): container finished" podID="ab7c2d8a-88e9-451c-938a-56a685d17a26" containerID="3ef8f22b0de05dec314ae50d025f4e10662b7ed893366d985e65a461a13dcbab" exitCode=60 Oct 30 18:15:04.894940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:04.894899 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" event={"ID":"ab7c2d8a-88e9-451c-938a-56a685d17a26","Type":"ContainerDied","Data":"3ef8f22b0de05dec314ae50d025f4e10662b7ed893366d985e65a461a13dcbab"} Oct 30 18:15:04.895381 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:04.895268 5232 scope.go:122] "RemoveContainer" containerID="3ef8f22b0de05dec314ae50d025f4e10662b7ed893366d985e65a461a13dcbab" Oct 30 18:15:04.896740 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:04.896719 5232 generic.go:356] "Generic (PLEG): container finished" podID="34bb7979-7eb6-4b2f-a7d2-90acc05254f4" containerID="fb8442e0f5ce226601d179c023d9e3bbdc44e5ff87b68f421b7990893bffcd13" exitCode=60 Oct 30 18:15:04.896877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:04.896749 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" event={"ID":"34bb7979-7eb6-4b2f-a7d2-90acc05254f4","Type":"ContainerDied","Data":"fb8442e0f5ce226601d179c023d9e3bbdc44e5ff87b68f421b7990893bffcd13"} Oct 30 18:15:04.897010 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:04.896997 5232 scope.go:122] "RemoveContainer" containerID="fb8442e0f5ce226601d179c023d9e3bbdc44e5ff87b68f421b7990893bffcd13" Oct 30 18:15:04.898621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:04.898598 5232 generic.go:356] "Generic (PLEG): container finished" podID="367e7ada-dcd6-47fc-9117-76a1d38c84ed" containerID="afeb77f71981109048f62d0cc142dbba711000707f1ec94b942b784e2fb0dcbf" exitCode=60 Oct 30 18:15:04.898747 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:04.898665 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" event={"ID":"367e7ada-dcd6-47fc-9117-76a1d38c84ed","Type":"ContainerDied","Data":"afeb77f71981109048f62d0cc142dbba711000707f1ec94b942b784e2fb0dcbf"} Oct 30 18:15:04.898877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:04.898864 5232 scope.go:122] "RemoveContainer" containerID="afeb77f71981109048f62d0cc142dbba711000707f1ec94b942b784e2fb0dcbf" Oct 30 18:15:04.900534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:04.900491 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" event={"ID":"3060b110-ab37-472b-b82a-c360d5bbd9a8","Type":"ContainerStarted","Data":"0b678bd367e7ac821518ebe104f8e56d6316900e12516853e405bf8d5756f22d"} Oct 30 18:15:04.902590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:04.902569 5232 generic.go:356] "Generic (PLEG): container finished" podID="44a3dd61-3a4f-40b9-8c5f-0896c37c5029" containerID="244d43976c3c2168b15ecff9f16131a979d51d45135b4ac22673116ab4504dd9" exitCode=60 Oct 30 18:15:04.902682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:04.902598 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" event={"ID":"44a3dd61-3a4f-40b9-8c5f-0896c37c5029","Type":"ContainerDied","Data":"244d43976c3c2168b15ecff9f16131a979d51d45135b4ac22673116ab4504dd9"} Oct 30 18:15:04.902801 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:04.902788 5232 scope.go:122] "RemoveContainer" containerID="244d43976c3c2168b15ecff9f16131a979d51d45135b4ac22673116ab4504dd9" Oct 30 18:15:05.131118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:05.131072 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" podStartSLOduration=5.131056675 podStartE2EDuration="5.131056675s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:05.13016203 +0000 UTC m=+804.446582352" watchObservedRunningTime="2026-10-30 18:15:05.131056675 +0000 UTC m=+804.447477019" Oct 30 18:15:05.913594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:05.913206 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" event={"ID":"44a3dd61-3a4f-40b9-8c5f-0896c37c5029","Type":"ContainerStarted","Data":"8d08aefcded6eeb55d267f947ca4999ef46e800dc2d9290f3d7f281ef836d859"} Oct 30 18:15:05.917095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:05.917066 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" event={"ID":"ab7c2d8a-88e9-451c-938a-56a685d17a26","Type":"ContainerStarted","Data":"2b2bbf0c59d4c8955a90cbf06c9554ff5f7a876786261bb4f5360809dd752e94"} Oct 30 18:15:05.920117 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:05.920091 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" event={"ID":"34bb7979-7eb6-4b2f-a7d2-90acc05254f4","Type":"ContainerStarted","Data":"bc6b601f36fc86556678549b6968d935e85bf8df66a915c5a548c2060420543d"} Oct 30 18:15:05.922432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:05.922404 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" event={"ID":"367e7ada-dcd6-47fc-9117-76a1d38c84ed","Type":"ContainerStarted","Data":"1fe67c31db3176d84f99c809db2b1d10f05eb8b73b86632ea6556771758eea91"} Oct 30 18:15:05.924943 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:05.924917 5232 generic.go:356] "Generic (PLEG): container finished" podID="3060b110-ab37-472b-b82a-c360d5bbd9a8" containerID="0b678bd367e7ac821518ebe104f8e56d6316900e12516853e405bf8d5756f22d" exitCode=60 Oct 30 18:15:05.925036 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:05.924997 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" event={"ID":"3060b110-ab37-472b-b82a-c360d5bbd9a8","Type":"ContainerDied","Data":"0b678bd367e7ac821518ebe104f8e56d6316900e12516853e405bf8d5756f22d"} Oct 30 18:15:05.925187 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:05.925174 5232 scope.go:122] "RemoveContainer" containerID="0b678bd367e7ac821518ebe104f8e56d6316900e12516853e405bf8d5756f22d" Oct 30 18:15:05.953967 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:05.953922 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" podStartSLOduration=5.953906199 podStartE2EDuration="5.953906199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:05.950449398 +0000 UTC m=+805.266869719" watchObservedRunningTime="2026-10-30 18:15:05.953906199 +0000 UTC m=+805.270326519" Oct 30 18:15:06.027707 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.027646 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" podStartSLOduration=6.027631589 podStartE2EDuration="6.027631589s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:06.025873374 +0000 UTC m=+805.342293706" watchObservedRunningTime="2026-10-30 18:15:06.027631589 +0000 UTC m=+805.344051910" Oct 30 18:15:06.066244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.066188 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" podStartSLOduration=6.066173219 podStartE2EDuration="6.066173219s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:06.063043401 +0000 UTC m=+805.379463722" watchObservedRunningTime="2026-10-30 18:15:06.066173219 +0000 UTC m=+805.382593580" Oct 30 18:15:06.104136 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.104072 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" podStartSLOduration=6.104050463 podStartE2EDuration="6.104050463s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:06.099461909 +0000 UTC m=+805.415882230" watchObservedRunningTime="2026-10-30 18:15:06.104050463 +0000 UTC m=+805.420470788" Oct 30 18:15:06.934358 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.934326 5232 generic.go:356] "Generic (PLEG): container finished" podID="367e7ada-dcd6-47fc-9117-76a1d38c84ed" containerID="1fe67c31db3176d84f99c809db2b1d10f05eb8b73b86632ea6556771758eea91" exitCode=60 Oct 30 18:15:06.934849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.934414 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" event={"ID":"367e7ada-dcd6-47fc-9117-76a1d38c84ed","Type":"ContainerDied","Data":"1fe67c31db3176d84f99c809db2b1d10f05eb8b73b86632ea6556771758eea91"} Oct 30 18:15:06.934849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.934482 5232 scope.go:122] "RemoveContainer" containerID="afeb77f71981109048f62d0cc142dbba711000707f1ec94b942b784e2fb0dcbf" Oct 30 18:15:06.934849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.934756 5232 scope.go:122] "RemoveContainer" containerID="1fe67c31db3176d84f99c809db2b1d10f05eb8b73b86632ea6556771758eea91" Oct 30 18:15:06.935140 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:15:06.935117 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2_odh-ai-gateway-infra(367e7ada-dcd6-47fc-9117-76a1d38c84ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" podUID="367e7ada-dcd6-47fc-9117-76a1d38c84ed" Oct 30 18:15:06.936450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.936427 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" event={"ID":"3060b110-ab37-472b-b82a-c360d5bbd9a8","Type":"ContainerStarted","Data":"69c1183b8828674429266ac557d20b08f92204353b188ac6c736a8d92b6a567f"} Oct 30 18:15:06.938294 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.938265 5232 generic.go:356] "Generic (PLEG): container finished" podID="44a3dd61-3a4f-40b9-8c5f-0896c37c5029" containerID="8d08aefcded6eeb55d267f947ca4999ef46e800dc2d9290f3d7f281ef836d859" exitCode=60 Oct 30 18:15:06.938417 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.938398 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" event={"ID":"44a3dd61-3a4f-40b9-8c5f-0896c37c5029","Type":"ContainerDied","Data":"8d08aefcded6eeb55d267f947ca4999ef46e800dc2d9290f3d7f281ef836d859"} Oct 30 18:15:06.938554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.938532 5232 scope.go:122] "RemoveContainer" containerID="8d08aefcded6eeb55d267f947ca4999ef46e800dc2d9290f3d7f281ef836d859" Oct 30 18:15:06.938790 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:15:06.938765 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6_odh-ai-gateway-infra(44a3dd61-3a4f-40b9-8c5f-0896c37c5029)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" podUID="44a3dd61-3a4f-40b9-8c5f-0896c37c5029" Oct 30 18:15:06.940436 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.940418 5232 generic.go:356] "Generic (PLEG): container finished" podID="ab7c2d8a-88e9-451c-938a-56a685d17a26" containerID="2b2bbf0c59d4c8955a90cbf06c9554ff5f7a876786261bb4f5360809dd752e94" exitCode=60 Oct 30 18:15:06.940553 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.940518 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" event={"ID":"ab7c2d8a-88e9-451c-938a-56a685d17a26","Type":"ContainerDied","Data":"2b2bbf0c59d4c8955a90cbf06c9554ff5f7a876786261bb4f5360809dd752e94"} Oct 30 18:15:06.940705 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.940690 5232 scope.go:122] "RemoveContainer" containerID="2b2bbf0c59d4c8955a90cbf06c9554ff5f7a876786261bb4f5360809dd752e94" Oct 30 18:15:06.940965 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:15:06.940944 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw_odh-ai-gateway-infra(ab7c2d8a-88e9-451c-938a-56a685d17a26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" podUID="ab7c2d8a-88e9-451c-938a-56a685d17a26" Oct 30 18:15:06.942398 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.942378 5232 generic.go:356] "Generic (PLEG): container finished" podID="34bb7979-7eb6-4b2f-a7d2-90acc05254f4" containerID="bc6b601f36fc86556678549b6968d935e85bf8df66a915c5a548c2060420543d" exitCode=60 Oct 30 18:15:06.942483 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.942424 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" event={"ID":"34bb7979-7eb6-4b2f-a7d2-90acc05254f4","Type":"ContainerDied","Data":"bc6b601f36fc86556678549b6968d935e85bf8df66a915c5a548c2060420543d"} Oct 30 18:15:06.942681 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.942660 5232 scope.go:122] "RemoveContainer" containerID="bc6b601f36fc86556678549b6968d935e85bf8df66a915c5a548c2060420543d" Oct 30 18:15:06.942899 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:15:06.942883 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v_odh-ai-gateway-infra(34bb7979-7eb6-4b2f-a7d2-90acc05254f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" podUID="34bb7979-7eb6-4b2f-a7d2-90acc05254f4" Oct 30 18:15:06.949231 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.949210 5232 scope.go:122] "RemoveContainer" containerID="244d43976c3c2168b15ecff9f16131a979d51d45135b4ac22673116ab4504dd9" Oct 30 18:15:06.964189 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.964156 5232 scope.go:122] "RemoveContainer" containerID="3ef8f22b0de05dec314ae50d025f4e10662b7ed893366d985e65a461a13dcbab" Oct 30 18:15:06.978270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:06.978241 5232 scope.go:122] "RemoveContainer" containerID="fb8442e0f5ce226601d179c023d9e3bbdc44e5ff87b68f421b7990893bffcd13" Oct 30 18:15:07.951352 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:07.951322 5232 scope.go:122] "RemoveContainer" containerID="8d08aefcded6eeb55d267f947ca4999ef46e800dc2d9290f3d7f281ef836d859" Oct 30 18:15:07.951806 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:15:07.951703 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6_odh-ai-gateway-infra(44a3dd61-3a4f-40b9-8c5f-0896c37c5029)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" podUID="44a3dd61-3a4f-40b9-8c5f-0896c37c5029" Oct 30 18:15:07.952902 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:07.952876 5232 scope.go:122] "RemoveContainer" containerID="2b2bbf0c59d4c8955a90cbf06c9554ff5f7a876786261bb4f5360809dd752e94" Oct 30 18:15:07.953194 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:15:07.953169 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw_odh-ai-gateway-infra(ab7c2d8a-88e9-451c-938a-56a685d17a26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" podUID="ab7c2d8a-88e9-451c-938a-56a685d17a26" Oct 30 18:15:07.954475 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:07.954459 5232 scope.go:122] "RemoveContainer" containerID="bc6b601f36fc86556678549b6968d935e85bf8df66a915c5a548c2060420543d" Oct 30 18:15:07.954749 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:15:07.954730 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v_odh-ai-gateway-infra(34bb7979-7eb6-4b2f-a7d2-90acc05254f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" podUID="34bb7979-7eb6-4b2f-a7d2-90acc05254f4" Oct 30 18:15:07.955916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:07.955902 5232 scope.go:122] "RemoveContainer" containerID="1fe67c31db3176d84f99c809db2b1d10f05eb8b73b86632ea6556771758eea91" Oct 30 18:15:07.956141 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:15:07.956124 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2_odh-ai-gateway-infra(367e7ada-dcd6-47fc-9117-76a1d38c84ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" podUID="367e7ada-dcd6-47fc-9117-76a1d38c84ed" Oct 30 18:15:07.957481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:07.957458 5232 generic.go:356] "Generic (PLEG): container finished" podID="3060b110-ab37-472b-b82a-c360d5bbd9a8" containerID="69c1183b8828674429266ac557d20b08f92204353b188ac6c736a8d92b6a567f" exitCode=60 Oct 30 18:15:07.957625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:07.957545 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" event={"ID":"3060b110-ab37-472b-b82a-c360d5bbd9a8","Type":"ContainerDied","Data":"69c1183b8828674429266ac557d20b08f92204353b188ac6c736a8d92b6a567f"} Oct 30 18:15:07.957625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:07.957621 5232 scope.go:122] "RemoveContainer" containerID="0b678bd367e7ac821518ebe104f8e56d6316900e12516853e405bf8d5756f22d" Oct 30 18:15:07.957819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:07.957803 5232 scope.go:122] "RemoveContainer" containerID="69c1183b8828674429266ac557d20b08f92204353b188ac6c736a8d92b6a567f" Oct 30 18:15:07.958040 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:15:07.958017 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb_odh-ai-gateway-infra(3060b110-ab37-472b-b82a-c360d5bbd9a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" podUID="3060b110-ab37-472b-b82a-c360d5bbd9a8" Oct 30 18:15:08.793766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.793727 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6"] Oct 30 18:15:08.799139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.799114 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.833625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.833592 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6"] Oct 30 18:15:08.835892 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.835865 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4"] Oct 30 18:15:08.840100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.840078 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.850301 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.848533 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4"] Oct 30 18:15:08.873844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.873808 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.874032 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.873850 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r66wj\" (UniqueName: \"kubernetes.io/projected/8a806882-2d84-427c-8698-9ed54994b748-kube-api-access-r66wj\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.874032 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.873877 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.874153 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.874002 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a806882-2d84-427c-8698-9ed54994b748-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.874153 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.874067 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.874153 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.874116 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.874153 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.874140 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.874359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.874161 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.874359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.874291 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.874359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.874312 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.874359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.874335 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.874581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.874368 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rqs8v\" (UniqueName: \"kubernetes.io/projected/897d6674-2313-4b21-8a84-d84d56d1c22e-kube-api-access-rqs8v\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.874581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.874438 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.874581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.874466 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a806882-2d84-427c-8698-9ed54994b748-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.874581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.874487 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.874581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.874547 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.874581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.874565 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/897d6674-2313-4b21-8a84-d84d56d1c22e-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.874890 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.874616 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a806882-2d84-427c-8698-9ed54994b748-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.966907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.966882 5232 scope.go:122] "RemoveContainer" containerID="69c1183b8828674429266ac557d20b08f92204353b188ac6c736a8d92b6a567f" Oct 30 18:15:08.967266 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:15:08.967177 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb_odh-ai-gateway-infra(3060b110-ab37-472b-b82a-c360d5bbd9a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" podUID="3060b110-ab37-472b-b82a-c360d5bbd9a8" Oct 30 18:15:08.975436 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.975409 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a806882-2d84-427c-8698-9ed54994b748-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.975567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.975447 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.975567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.975478 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.975567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.975502 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/897d6674-2313-4b21-8a84-d84d56d1c22e-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.975740 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.975587 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a806882-2d84-427c-8698-9ed54994b748-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.975740 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.975707 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.975851 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.975750 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r66wj\" (UniqueName: \"kubernetes.io/projected/8a806882-2d84-427c-8698-9ed54994b748-kube-api-access-r66wj\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.975851 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.975779 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.975851 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.975817 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a806882-2d84-427c-8698-9ed54994b748-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.976008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.975864 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.976008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.975907 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.976008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.975919 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.976008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.975931 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.976008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.975962 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.976008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.975965 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.976638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.976117 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.976638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.976178 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.976638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.976245 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.976638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.976251 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.976638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.976447 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/897d6674-2313-4b21-8a84-d84d56d1c22e-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.976638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.976485 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.976638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.976563 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rqs8v\" (UniqueName: \"kubernetes.io/projected/897d6674-2313-4b21-8a84-d84d56d1c22e-kube-api-access-rqs8v\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.976638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.976608 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.977084 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.976797 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a806882-2d84-427c-8698-9ed54994b748-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.977084 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.976959 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.977180 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.977142 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.977250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.977172 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.977302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.977263 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.978346 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.978321 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.978890 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.978870 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a806882-2d84-427c-8698-9ed54994b748-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.979282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.979260 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.979553 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.979496 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.994730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.994698 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.994854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.994821 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rqs8v\" (UniqueName: \"kubernetes.io/projected/897d6674-2313-4b21-8a84-d84d56d1c22e-kube-api-access-rqs8v\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:08.995291 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.995266 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r66wj\" (UniqueName: \"kubernetes.io/projected/8a806882-2d84-427c-8698-9ed54994b748-kube-api-access-r66wj\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:08.995466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:08.995450 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a806882-2d84-427c-8698-9ed54994b748-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:09.128621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:09.128519 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:09.158695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:09.158662 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:09.446804 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:09.446779 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6"] Oct 30 18:15:09.448542 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:15:09.448484 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod897d6674_2313_4b21_8a84_d84d56d1c22e.slice/crio-3654064f6eedbf5018af6935cd037adeb53be0650830145fba4857ac407e86a3 WatchSource:0}: Error finding container 3654064f6eedbf5018af6935cd037adeb53be0650830145fba4857ac407e86a3: Status 404 returned error can't find the container with id 3654064f6eedbf5018af6935cd037adeb53be0650830145fba4857ac407e86a3 Oct 30 18:15:09.451217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:09.451178 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:15:09.451316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:09.451264 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:15:09.451316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:09.451307 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:15:09.482818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:09.482790 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4"] Oct 30 18:15:09.487848 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:15:09.487824 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a806882_2d84_427c_8698_9ed54994b748.slice/crio-9103bf0638175aaf6b18f0d7525b2fe6df92c53460b04a5974d30bee0bc1b197 WatchSource:0}: Error finding container 9103bf0638175aaf6b18f0d7525b2fe6df92c53460b04a5974d30bee0bc1b197: Status 404 returned error can't find the container with id 9103bf0638175aaf6b18f0d7525b2fe6df92c53460b04a5974d30bee0bc1b197 Oct 30 18:15:09.490318 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:09.490283 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:15:09.490415 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:09.490363 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:15:09.490476 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:09.490410 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:15:09.975296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:09.975259 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" event={"ID":"897d6674-2313-4b21-8a84-d84d56d1c22e","Type":"ContainerStarted","Data":"88b8a3bed368fe3fc9a323daf6ab1b1babcc75f691270e1c60b49f2c8155e453"} Oct 30 18:15:09.975773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:09.975326 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" event={"ID":"897d6674-2313-4b21-8a84-d84d56d1c22e","Type":"ContainerStarted","Data":"3654064f6eedbf5018af6935cd037adeb53be0650830145fba4857ac407e86a3"} Oct 30 18:15:09.977159 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:09.977129 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" event={"ID":"8a806882-2d84-427c-8698-9ed54994b748","Type":"ContainerStarted","Data":"197ba80f6b5e2c19aac5628d61b3f598aefe816d37e1b324df018c5dacf4cd4b"} Oct 30 18:15:09.977269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:09.977180 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" event={"ID":"8a806882-2d84-427c-8698-9ed54994b748","Type":"ContainerStarted","Data":"9103bf0638175aaf6b18f0d7525b2fe6df92c53460b04a5974d30bee0bc1b197"} Oct 30 18:15:10.021367 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:10.021298 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" podStartSLOduration=2.021276791 podStartE2EDuration="2.021276791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:10.018608459 +0000 UTC m=+809.335028781" watchObservedRunningTime="2026-10-30 18:15:10.021276791 +0000 UTC m=+809.337697113" Oct 30 18:15:10.065878 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:10.065819 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" podStartSLOduration=2.065804747 podStartE2EDuration="2.065804747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:10.061382158 +0000 UTC m=+809.377802478" watchObservedRunningTime="2026-10-30 18:15:10.065804747 +0000 UTC m=+809.382225068" Oct 30 18:15:10.129405 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:10.129373 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:10.135008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:10.134974 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:10.135258 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:10.135231 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" podUID="897d6674-2313-4b21-8a84-d84d56d1c22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 18:15:10.163364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:10.161840 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:10.170029 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:10.169992 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:15:10.170391 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:10.170327 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" podUID="8a806882-2d84-427c-8698-9ed54994b748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 18:15:11.133009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:11.132970 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:11.153586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:11.133592 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" podUID="897d6674-2313-4b21-8a84-d84d56d1c22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 18:15:11.161955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:11.161920 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:15:11.162203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:11.162175 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" podUID="8a806882-2d84-427c-8698-9ed54994b748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 18:15:12.146538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:12.145921 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:12.146538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:12.145980 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" podUID="897d6674-2313-4b21-8a84-d84d56d1c22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 18:15:12.161085 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:12.161048 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:15:12.161373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:12.161331 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" podUID="8a806882-2d84-427c-8698-9ed54994b748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 18:15:13.130081 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:13.130041 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:13.130375 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:13.130346 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" podUID="897d6674-2313-4b21-8a84-d84d56d1c22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 18:15:13.159668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:13.159614 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:15:13.160317 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:13.160266 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" podUID="8a806882-2d84-427c-8698-9ed54994b748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 18:15:14.131541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:14.131052 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:14.131541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:14.131124 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" podUID="897d6674-2313-4b21-8a84-d84d56d1c22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 18:15:14.165356 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:14.164906 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:15:14.165356 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:14.164984 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" podUID="8a806882-2d84-427c-8698-9ed54994b748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 18:15:15.129984 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:15.129939 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:15.130295 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:15.130265 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" podUID="897d6674-2313-4b21-8a84-d84d56d1c22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 18:15:15.160642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:15.160396 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:15:15.160642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:15.160502 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" podUID="8a806882-2d84-427c-8698-9ed54994b748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 18:15:16.129239 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:16.129199 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" start-of-body= Oct 30 18:15:16.129674 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:16.129271 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" podUID="897d6674-2313-4b21-8a84-d84d56d1c22e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.88:15021/healthz/ready\": dial tcp 10.133.0.88:15021: connect: connection refused" Oct 30 18:15:16.159855 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:16.159810 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" start-of-body= Oct 30 18:15:16.160025 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:16.159876 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" podUID="8a806882-2d84-427c-8698-9ed54994b748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": dial tcp 10.133.0.89:15021: connect: connection refused" Oct 30 18:15:16.795539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:16.795480 5232 scope.go:122] "RemoveContainer" containerID="2b2bbf0c59d4c8955a90cbf06c9554ff5f7a876786261bb4f5360809dd752e94" Oct 30 18:15:16.795720 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:16.795639 5232 scope.go:122] "RemoveContainer" containerID="bc6b601f36fc86556678549b6968d935e85bf8df66a915c5a548c2060420543d" Oct 30 18:15:17.133903 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:17.133873 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:17.134413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:17.134392 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:17.135234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:17.135214 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:15:17.165420 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:17.165388 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:17.165664 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:17.165632 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:17.166281 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:17.166261 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:15:17.979455 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:17.979371 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7"] Oct 30 18:15:17.984788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:17.984766 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.003295 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.003270 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7"] Oct 30 18:15:18.009628 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.009590 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"default-dockercfg-45pmf\"" Oct 30 18:15:18.011563 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.011539 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"kube-root-ca.crt\"" Oct 30 18:15:18.011563 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.011547 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"distinct-2-w6-ef13cc-kserve-self-signed-certs\"" Oct 30 18:15:18.011797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.011778 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-ef13cc\"/\"openshift-service-ca.crt\"" Oct 30 18:15:18.108824 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.108793 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-tls-certs\") pod \"distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.108975 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.108915 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-tmp-dir\") pod \"distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.108975 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.108960 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzkq9\" (UniqueName: \"kubernetes.io/projected/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-kube-api-access-lzkq9\") pod \"distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.109041 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.109026 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-home\") pod \"distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.109127 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.109115 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-model-cache\") pod \"distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.109160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.109150 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-dshm\") pod \"distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.123828 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.123790 5232 generic.go:356] "Generic (PLEG): container finished" podID="ab7c2d8a-88e9-451c-938a-56a685d17a26" containerID="ffa767eb8a7a8ad1ad0e42be1c9be7562e1da17374d3d8fa738bc731069711a1" exitCode=60 Oct 30 18:15:18.124009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.123863 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" event={"ID":"ab7c2d8a-88e9-451c-938a-56a685d17a26","Type":"ContainerDied","Data":"ffa767eb8a7a8ad1ad0e42be1c9be7562e1da17374d3d8fa738bc731069711a1"} Oct 30 18:15:18.124009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.123928 5232 scope.go:122] "RemoveContainer" containerID="2b2bbf0c59d4c8955a90cbf06c9554ff5f7a876786261bb4f5360809dd752e94" Oct 30 18:15:18.124366 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.124347 5232 scope.go:122] "RemoveContainer" containerID="ffa767eb8a7a8ad1ad0e42be1c9be7562e1da17374d3d8fa738bc731069711a1" Oct 30 18:15:18.124714 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:15:18.124688 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw_odh-ai-gateway-infra(ab7c2d8a-88e9-451c-938a-56a685d17a26)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" podUID="ab7c2d8a-88e9-451c-938a-56a685d17a26" Oct 30 18:15:18.126228 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.126208 5232 generic.go:356] "Generic (PLEG): container finished" podID="34bb7979-7eb6-4b2f-a7d2-90acc05254f4" containerID="5bf2c019091a8e112b450982983b1d791447bbe89113bd695c38333db5308c47" exitCode=60 Oct 30 18:15:18.126305 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.126280 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" event={"ID":"34bb7979-7eb6-4b2f-a7d2-90acc05254f4","Type":"ContainerDied","Data":"5bf2c019091a8e112b450982983b1d791447bbe89113bd695c38333db5308c47"} Oct 30 18:15:18.126851 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.126832 5232 scope.go:122] "RemoveContainer" containerID="5bf2c019091a8e112b450982983b1d791447bbe89113bd695c38333db5308c47" Oct 30 18:15:18.127067 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:15:18.127051 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v_odh-ai-gateway-infra(34bb7979-7eb6-4b2f-a7d2-90acc05254f4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" podUID="34bb7979-7eb6-4b2f-a7d2-90acc05254f4" Oct 30 18:15:18.140028 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.139999 5232 scope.go:122] "RemoveContainer" containerID="bc6b601f36fc86556678549b6968d935e85bf8df66a915c5a548c2060420543d" Oct 30 18:15:18.213533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.212856 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-tls-certs\") pod \"distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.213533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.213018 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-tmp-dir\") pod \"distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.213533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.213113 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzkq9\" (UniqueName: \"kubernetes.io/projected/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-kube-api-access-lzkq9\") pod \"distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.213533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.213263 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-home\") pod \"distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.213533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.213393 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-model-cache\") pod \"distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.213533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.213473 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-dshm\") pod \"distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.241528 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.238543 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-tmp-dir\") pod \"distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.241528 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.239765 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-home\") pod \"distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.241755 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.241721 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-model-cache\") pod \"distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.280551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.260787 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-dshm\") pod \"distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.289928 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.286221 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzkq9\" (UniqueName: \"kubernetes.io/projected/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-kube-api-access-lzkq9\") pod \"distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.289928 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.286779 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-tls-certs\") pod \"distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.310810 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.310771 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:18.667666 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.667638 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7"] Oct 30 18:15:18.668919 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:15:18.668894 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf7f7ebe4_e24c_4cd7_aa13_978eb635e01e.slice/crio-c152ccf12ab6f76ade57d01a44df17d91badb5c89437ea98c6c4ff7df7345116 WatchSource:0}: Error finding container c152ccf12ab6f76ade57d01a44df17d91badb5c89437ea98c6c4ff7df7345116: Status 404 returned error can't find the container with id c152ccf12ab6f76ade57d01a44df17d91badb5c89437ea98c6c4ff7df7345116 Oct 30 18:15:18.796210 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:18.796183 5232 scope.go:122] "RemoveContainer" containerID="1fe67c31db3176d84f99c809db2b1d10f05eb8b73b86632ea6556771758eea91" Oct 30 18:15:19.137315 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.137284 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" event={"ID":"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e","Type":"ContainerStarted","Data":"0e66854e85077b884fdf444a3ae0d64f02f9bcc5f5f7397a4f02dceec67afee9"} Oct 30 18:15:19.137481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.137328 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" event={"ID":"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e","Type":"ContainerStarted","Data":"c152ccf12ab6f76ade57d01a44df17d91badb5c89437ea98c6c4ff7df7345116"} Oct 30 18:15:19.140655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.140626 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" event={"ID":"367e7ada-dcd6-47fc-9117-76a1d38c84ed","Type":"ContainerStarted","Data":"b47160d03774bcfde5dff77c7c8af524c03163b8d761f17a46245c8645be4cf8"} Oct 30 18:15:19.173569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.173533 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw"] Oct 30 18:15:19.182122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.182069 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podStartSLOduration=2.182053788 podStartE2EDuration="2.182053788s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:19.179447155 +0000 UTC m=+818.495867505" watchObservedRunningTime="2026-10-30 18:15:19.182053788 +0000 UTC m=+818.498474136" Oct 30 18:15:19.301438 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.301356 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v"] Oct 30 18:15:19.311858 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.311821 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:19.351964 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.351928 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" Oct 30 18:15:19.485207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.437194 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab7c2d8a-88e9-451c-938a-56a685d17a26-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7c2d8a-88e9-451c-938a-56a685d17a26-maas-api-service-ca\") pod \"ab7c2d8a-88e9-451c-938a-56a685d17a26\" (UID: \"ab7c2d8a-88e9-451c-938a-56a685d17a26\") " Oct 30 18:15:19.485207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.437395 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab7c2d8a-88e9-451c-938a-56a685d17a26-kube-api-access-pftg8\" (UniqueName: \"kubernetes.io/projected/ab7c2d8a-88e9-451c-938a-56a685d17a26-kube-api-access-pftg8\") pod \"ab7c2d8a-88e9-451c-938a-56a685d17a26\" (UID: \"ab7c2d8a-88e9-451c-938a-56a685d17a26\") " Oct 30 18:15:19.485207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.443698 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab7c2d8a-88e9-451c-938a-56a685d17a26-maas-api-service-ca" pod "ab7c2d8a-88e9-451c-938a-56a685d17a26" (UID: "ab7c2d8a-88e9-451c-938a-56a685d17a26"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:19.485207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.475410 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab7c2d8a-88e9-451c-938a-56a685d17a26-maas-api-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:15:19.514001 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.513763 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab7c2d8a-88e9-451c-938a-56a685d17a26-kube-api-access-pftg8" pod "ab7c2d8a-88e9-451c-938a-56a685d17a26" (UID: "ab7c2d8a-88e9-451c-938a-56a685d17a26"). InnerVolumeSpecName "kube-api-access-pftg8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:19.541935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.541909 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" Oct 30 18:15:19.576532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.576417 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34bb7979-7eb6-4b2f-a7d2-90acc05254f4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34bb7979-7eb6-4b2f-a7d2-90acc05254f4-maas-api-service-ca\") pod \"34bb7979-7eb6-4b2f-a7d2-90acc05254f4\" (UID: \"34bb7979-7eb6-4b2f-a7d2-90acc05254f4\") " Oct 30 18:15:19.576692 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.576598 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34bb7979-7eb6-4b2f-a7d2-90acc05254f4-kube-api-access-g4stz\" (UniqueName: \"kubernetes.io/projected/34bb7979-7eb6-4b2f-a7d2-90acc05254f4-kube-api-access-g4stz\") pod \"34bb7979-7eb6-4b2f-a7d2-90acc05254f4\" (UID: \"34bb7979-7eb6-4b2f-a7d2-90acc05254f4\") " Oct 30 18:15:19.576817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.576794 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34bb7979-7eb6-4b2f-a7d2-90acc05254f4-maas-api-service-ca" pod "34bb7979-7eb6-4b2f-a7d2-90acc05254f4" (UID: "34bb7979-7eb6-4b2f-a7d2-90acc05254f4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:19.577135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.577115 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/34bb7979-7eb6-4b2f-a7d2-90acc05254f4-maas-api-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:15:19.577213 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.577140 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pftg8\" (UniqueName: \"kubernetes.io/projected/ab7c2d8a-88e9-451c-938a-56a685d17a26-kube-api-access-pftg8\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:15:19.578808 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.578785 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34bb7979-7eb6-4b2f-a7d2-90acc05254f4-kube-api-access-g4stz" pod "34bb7979-7eb6-4b2f-a7d2-90acc05254f4" (UID: "34bb7979-7eb6-4b2f-a7d2-90acc05254f4"). InnerVolumeSpecName "kube-api-access-g4stz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:19.678548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.678497 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g4stz\" (UniqueName: \"kubernetes.io/projected/34bb7979-7eb6-4b2f-a7d2-90acc05254f4-kube-api-access-g4stz\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:15:19.796183 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:19.796153 5232 scope.go:122] "RemoveContainer" containerID="8d08aefcded6eeb55d267f947ca4999ef46e800dc2d9290f3d7f281ef836d859" Oct 30 18:15:20.154595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.154493 5232 generic.go:356] "Generic (PLEG): container finished" podID="367e7ada-dcd6-47fc-9117-76a1d38c84ed" containerID="b47160d03774bcfde5dff77c7c8af524c03163b8d761f17a46245c8645be4cf8" exitCode=60 Oct 30 18:15:20.154595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.154539 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" event={"ID":"367e7ada-dcd6-47fc-9117-76a1d38c84ed","Type":"ContainerDied","Data":"b47160d03774bcfde5dff77c7c8af524c03163b8d761f17a46245c8645be4cf8"} Oct 30 18:15:20.155091 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.154602 5232 scope.go:122] "RemoveContainer" containerID="1fe67c31db3176d84f99c809db2b1d10f05eb8b73b86632ea6556771758eea91" Oct 30 18:15:20.155091 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.154995 5232 scope.go:122] "RemoveContainer" containerID="b47160d03774bcfde5dff77c7c8af524c03163b8d761f17a46245c8645be4cf8" Oct 30 18:15:20.155359 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:15:20.155338 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2_odh-ai-gateway-infra(367e7ada-dcd6-47fc-9117-76a1d38c84ed)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" podUID="367e7ada-dcd6-47fc-9117-76a1d38c84ed" Oct 30 18:15:20.156925 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.156899 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" event={"ID":"44a3dd61-3a4f-40b9-8c5f-0896c37c5029","Type":"ContainerStarted","Data":"66921d5f2dca37ca647cb2fc5d4b2fa4a40401269035f2624eb395cea0763d4c"} Oct 30 18:15:20.158466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.158445 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" Oct 30 18:15:20.158982 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.158480 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw" event={"ID":"ab7c2d8a-88e9-451c-938a-56a685d17a26","Type":"ContainerDied","Data":"d3e2b6a42d47d5151bb1c388068bff2fd2341fb493d91ce1bcf3952e3b8be66d"} Oct 30 18:15:20.161685 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.161587 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" Oct 30 18:15:20.161685 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.161630 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v" event={"ID":"34bb7979-7eb6-4b2f-a7d2-90acc05254f4","Type":"ContainerDied","Data":"2277aff8c418e160108965a174f6363a3b5e7254dd47671965b1ae59945f3f91"} Oct 30 18:15:20.169148 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.169127 5232 scope.go:122] "RemoveContainer" containerID="ffa767eb8a7a8ad1ad0e42be1c9be7562e1da17374d3d8fa738bc731069711a1" Oct 30 18:15:20.180065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.179888 5232 scope.go:122] "RemoveContainer" containerID="5bf2c019091a8e112b450982983b1d791447bbe89113bd695c38333db5308c47" Oct 30 18:15:20.231123 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.231088 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v"] Oct 30 18:15:20.236587 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.236559 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889735-kzl2v"] Oct 30 18:15:20.270231 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.270203 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2"] Oct 30 18:15:20.274170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.274145 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw"] Oct 30 18:15:20.279952 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.279926 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-ef13cc-29889735-twcqw"] Oct 30 18:15:20.800767 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.800742 5232 scope.go:122] "RemoveContainer" containerID="69c1183b8828674429266ac557d20b08f92204353b188ac6c736a8d92b6a567f" Oct 30 18:15:20.806670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.806635 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34bb7979-7eb6-4b2f-a7d2-90acc05254f4" path="/var/lib/kubelet/pods/34bb7979-7eb6-4b2f-a7d2-90acc05254f4/volumes" Oct 30 18:15:20.807338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:20.807312 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab7c2d8a-88e9-451c-938a-56a685d17a26" path="/var/lib/kubelet/pods/ab7c2d8a-88e9-451c-938a-56a685d17a26/volumes" Oct 30 18:15:21.185537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:21.175802 5232 generic.go:356] "Generic (PLEG): container finished" podID="44a3dd61-3a4f-40b9-8c5f-0896c37c5029" containerID="66921d5f2dca37ca647cb2fc5d4b2fa4a40401269035f2624eb395cea0763d4c" exitCode=60 Oct 30 18:15:21.185537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:21.175980 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" event={"ID":"44a3dd61-3a4f-40b9-8c5f-0896c37c5029","Type":"ContainerDied","Data":"66921d5f2dca37ca647cb2fc5d4b2fa4a40401269035f2624eb395cea0763d4c"} Oct 30 18:15:21.185537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:21.176035 5232 scope.go:122] "RemoveContainer" containerID="8d08aefcded6eeb55d267f947ca4999ef46e800dc2d9290f3d7f281ef836d859" Oct 30 18:15:21.185537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:21.176622 5232 scope.go:122] "RemoveContainer" containerID="66921d5f2dca37ca647cb2fc5d4b2fa4a40401269035f2624eb395cea0763d4c" Oct 30 18:15:21.185537 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:15:21.176982 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6_odh-ai-gateway-infra(44a3dd61-3a4f-40b9-8c5f-0896c37c5029)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" podUID="44a3dd61-3a4f-40b9-8c5f-0896c37c5029" Oct 30 18:15:21.342578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:21.338874 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6"] Oct 30 18:15:21.572248 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:21.572222 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" Oct 30 18:15:21.612708 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:21.612605 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/367e7ada-dcd6-47fc-9117-76a1d38c84ed-kube-api-access-b5fwz\" (UniqueName: \"kubernetes.io/projected/367e7ada-dcd6-47fc-9117-76a1d38c84ed-kube-api-access-b5fwz\") pod \"367e7ada-dcd6-47fc-9117-76a1d38c84ed\" (UID: \"367e7ada-dcd6-47fc-9117-76a1d38c84ed\") " Oct 30 18:15:21.612892 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:21.612728 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/367e7ada-dcd6-47fc-9117-76a1d38c84ed-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/367e7ada-dcd6-47fc-9117-76a1d38c84ed-maas-api-service-ca\") pod \"367e7ada-dcd6-47fc-9117-76a1d38c84ed\" (UID: \"367e7ada-dcd6-47fc-9117-76a1d38c84ed\") " Oct 30 18:15:21.613904 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:21.613724 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/367e7ada-dcd6-47fc-9117-76a1d38c84ed-maas-api-service-ca" pod "367e7ada-dcd6-47fc-9117-76a1d38c84ed" (UID: "367e7ada-dcd6-47fc-9117-76a1d38c84ed"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:21.616026 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:21.615997 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/367e7ada-dcd6-47fc-9117-76a1d38c84ed-kube-api-access-b5fwz" pod "367e7ada-dcd6-47fc-9117-76a1d38c84ed" (UID: "367e7ada-dcd6-47fc-9117-76a1d38c84ed"). InnerVolumeSpecName "kube-api-access-b5fwz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:21.714827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:21.714790 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/367e7ada-dcd6-47fc-9117-76a1d38c84ed-maas-api-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:15:21.714827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:21.714820 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b5fwz\" (UniqueName: \"kubernetes.io/projected/367e7ada-dcd6-47fc-9117-76a1d38c84ed-kube-api-access-b5fwz\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.198625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:22.198600 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" Oct 30 18:15:22.198987 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:22.198602 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2" event={"ID":"367e7ada-dcd6-47fc-9117-76a1d38c84ed","Type":"ContainerDied","Data":"77d0406acc985f0c434d9dd07199c99f84f08bef1e8df536a0c56bd7b1df56d3"} Oct 30 18:15:22.198987 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:22.198732 5232 scope.go:122] "RemoveContainer" containerID="b47160d03774bcfde5dff77c7c8af524c03163b8d761f17a46245c8645be4cf8" Oct 30 18:15:22.201130 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:22.201103 5232 generic.go:356] "Generic (PLEG): container finished" podID="3060b110-ab37-472b-b82a-c360d5bbd9a8" containerID="1a55881cf040815c190142b13f7ef72df32e2cc40fcb81cf9d82ab8d021813d8" exitCode=60 Oct 30 18:15:22.201242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:22.201179 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" event={"ID":"3060b110-ab37-472b-b82a-c360d5bbd9a8","Type":"ContainerDied","Data":"1a55881cf040815c190142b13f7ef72df32e2cc40fcb81cf9d82ab8d021813d8"} Oct 30 18:15:22.201646 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:22.201630 5232 scope.go:122] "RemoveContainer" containerID="1a55881cf040815c190142b13f7ef72df32e2cc40fcb81cf9d82ab8d021813d8" Oct 30 18:15:22.201940 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:15:22.201920 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb_odh-ai-gateway-infra(3060b110-ab37-472b-b82a-c360d5bbd9a8)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" podUID="3060b110-ab37-472b-b82a-c360d5bbd9a8" Oct 30 18:15:22.217426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:22.217401 5232 scope.go:122] "RemoveContainer" containerID="69c1183b8828674429266ac557d20b08f92204353b188ac6c736a8d92b6a567f" Oct 30 18:15:22.283702 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:22.282612 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2"] Oct 30 18:15:22.284147 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:22.284047 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-6ce5eb-29889735-b5kv2"] Oct 30 18:15:22.354207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:22.354183 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" Oct 30 18:15:22.425830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:22.425793 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44a3dd61-3a4f-40b9-8c5f-0896c37c5029-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44a3dd61-3a4f-40b9-8c5f-0896c37c5029-maas-api-service-ca\") pod \"44a3dd61-3a4f-40b9-8c5f-0896c37c5029\" (UID: \"44a3dd61-3a4f-40b9-8c5f-0896c37c5029\") " Oct 30 18:15:22.426004 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:22.425971 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44a3dd61-3a4f-40b9-8c5f-0896c37c5029-kube-api-access-whr5w\" (UniqueName: \"kubernetes.io/projected/44a3dd61-3a4f-40b9-8c5f-0896c37c5029-kube-api-access-whr5w\") pod \"44a3dd61-3a4f-40b9-8c5f-0896c37c5029\" (UID: \"44a3dd61-3a4f-40b9-8c5f-0896c37c5029\") " Oct 30 18:15:22.426249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:22.426222 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44a3dd61-3a4f-40b9-8c5f-0896c37c5029-maas-api-service-ca" pod "44a3dd61-3a4f-40b9-8c5f-0896c37c5029" (UID: "44a3dd61-3a4f-40b9-8c5f-0896c37c5029"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:22.426654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:22.426628 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/44a3dd61-3a4f-40b9-8c5f-0896c37c5029-maas-api-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.428280 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:22.428255 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44a3dd61-3a4f-40b9-8c5f-0896c37c5029-kube-api-access-whr5w" pod "44a3dd61-3a4f-40b9-8c5f-0896c37c5029" (UID: "44a3dd61-3a4f-40b9-8c5f-0896c37c5029"). InnerVolumeSpecName "kube-api-access-whr5w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:22.527759 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:22.527725 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-whr5w\" (UniqueName: \"kubernetes.io/projected/44a3dd61-3a4f-40b9-8c5f-0896c37c5029-kube-api-access-whr5w\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:15:22.871472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:22.871380 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="367e7ada-dcd6-47fc-9117-76a1d38c84ed" path="/var/lib/kubelet/pods/367e7ada-dcd6-47fc-9117-76a1d38c84ed/volumes" Oct 30 18:15:23.212577 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.212470 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" Oct 30 18:15:23.212577 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.212477 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6" event={"ID":"44a3dd61-3a4f-40b9-8c5f-0896c37c5029","Type":"ContainerDied","Data":"4037551a184962e299564d38298a05c030733eeac84338e1407ee187835107b9"} Oct 30 18:15:23.212577 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.212570 5232 scope.go:122] "RemoveContainer" containerID="66921d5f2dca37ca647cb2fc5d4b2fa4a40401269035f2624eb395cea0763d4c" Oct 30 18:15:23.265309 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.265266 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6"] Oct 30 18:15:23.276722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.276699 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-fdf4da-29889735-lwcs6"] Oct 30 18:15:23.329030 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.329005 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb"] Oct 30 18:15:23.484318 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.484296 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" Oct 30 18:15:23.543964 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.543932 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3060b110-ab37-472b-b82a-c360d5bbd9a8-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3060b110-ab37-472b-b82a-c360d5bbd9a8-maas-api-service-ca\") pod \"3060b110-ab37-472b-b82a-c360d5bbd9a8\" (UID: \"3060b110-ab37-472b-b82a-c360d5bbd9a8\") " Oct 30 18:15:23.544172 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.544043 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3060b110-ab37-472b-b82a-c360d5bbd9a8-kube-api-access-czw9z\" (UniqueName: \"kubernetes.io/projected/3060b110-ab37-472b-b82a-c360d5bbd9a8-kube-api-access-czw9z\") pod \"3060b110-ab37-472b-b82a-c360d5bbd9a8\" (UID: \"3060b110-ab37-472b-b82a-c360d5bbd9a8\") " Oct 30 18:15:23.544407 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.544378 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3060b110-ab37-472b-b82a-c360d5bbd9a8-maas-api-service-ca" pod "3060b110-ab37-472b-b82a-c360d5bbd9a8" (UID: "3060b110-ab37-472b-b82a-c360d5bbd9a8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:23.544594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.544577 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3060b110-ab37-472b-b82a-c360d5bbd9a8-maas-api-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:15:23.546992 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.546963 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3060b110-ab37-472b-b82a-c360d5bbd9a8-kube-api-access-czw9z" pod "3060b110-ab37-472b-b82a-c360d5bbd9a8" (UID: "3060b110-ab37-472b-b82a-c360d5bbd9a8"). InnerVolumeSpecName "kube-api-access-czw9z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:23.645893 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.645861 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czw9z\" (UniqueName: \"kubernetes.io/projected/3060b110-ab37-472b-b82a-c360d5bbd9a8-kube-api-access-czw9z\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:15:23.798215 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798181 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj"] Oct 30 18:15:23.798806 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798790 5232 cpu_manager.go:414] "containerMap: removing container" podUID="367e7ada-dcd6-47fc-9117-76a1d38c84ed" containerName="cleanup" Oct 30 18:15:23.798856 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798807 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367e7ada-dcd6-47fc-9117-76a1d38c84ed" containerName="cleanup" Oct 30 18:15:23.798856 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798816 5232 cpu_manager.go:414] "containerMap: removing container" podUID="ab7c2d8a-88e9-451c-938a-56a685d17a26" containerName="cleanup" Oct 30 18:15:23.798856 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798821 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7c2d8a-88e9-451c-938a-56a685d17a26" containerName="cleanup" Oct 30 18:15:23.798856 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798827 5232 cpu_manager.go:414] "containerMap: removing container" podUID="44a3dd61-3a4f-40b9-8c5f-0896c37c5029" containerName="cleanup" Oct 30 18:15:23.798856 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798831 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a3dd61-3a4f-40b9-8c5f-0896c37c5029" containerName="cleanup" Oct 30 18:15:23.798856 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798843 5232 cpu_manager.go:414] "containerMap: removing container" podUID="ab7c2d8a-88e9-451c-938a-56a685d17a26" containerName="cleanup" Oct 30 18:15:23.798856 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798847 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7c2d8a-88e9-451c-938a-56a685d17a26" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798861 5232 cpu_manager.go:414] "containerMap: removing container" podUID="44a3dd61-3a4f-40b9-8c5f-0896c37c5029" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798865 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a3dd61-3a4f-40b9-8c5f-0896c37c5029" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798880 5232 cpu_manager.go:414] "containerMap: removing container" podUID="367e7ada-dcd6-47fc-9117-76a1d38c84ed" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798885 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367e7ada-dcd6-47fc-9117-76a1d38c84ed" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798892 5232 cpu_manager.go:414] "containerMap: removing container" podUID="34bb7979-7eb6-4b2f-a7d2-90acc05254f4" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798896 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34bb7979-7eb6-4b2f-a7d2-90acc05254f4" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798904 5232 cpu_manager.go:414] "containerMap: removing container" podUID="3060b110-ab37-472b-b82a-c360d5bbd9a8" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798907 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3060b110-ab37-472b-b82a-c360d5bbd9a8" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798916 5232 cpu_manager.go:414] "containerMap: removing container" podUID="3060b110-ab37-472b-b82a-c360d5bbd9a8" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798919 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3060b110-ab37-472b-b82a-c360d5bbd9a8" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798926 5232 cpu_manager.go:414] "containerMap: removing container" podUID="34bb7979-7eb6-4b2f-a7d2-90acc05254f4" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798930 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34bb7979-7eb6-4b2f-a7d2-90acc05254f4" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798940 5232 cpu_manager.go:414] "containerMap: removing container" podUID="34bb7979-7eb6-4b2f-a7d2-90acc05254f4" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.798944 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34bb7979-7eb6-4b2f-a7d2-90acc05254f4" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.799021 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="367e7ada-dcd6-47fc-9117-76a1d38c84ed" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.799027 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7c2d8a-88e9-451c-938a-56a685d17a26" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.799036 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a3dd61-3a4f-40b9-8c5f-0896c37c5029" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.799042 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7c2d8a-88e9-451c-938a-56a685d17a26" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.799050 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab7c2d8a-88e9-451c-938a-56a685d17a26" containerName="cleanup" Oct 30 18:15:23.799052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.799057 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="367e7ada-dcd6-47fc-9117-76a1d38c84ed" containerName="cleanup" Oct 30 18:15:23.799609 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.799062 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="34bb7979-7eb6-4b2f-a7d2-90acc05254f4" containerName="cleanup" Oct 30 18:15:23.799609 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.799067 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="3060b110-ab37-472b-b82a-c360d5bbd9a8" containerName="cleanup" Oct 30 18:15:23.799609 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.799072 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="3060b110-ab37-472b-b82a-c360d5bbd9a8" containerName="cleanup" Oct 30 18:15:23.799609 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.799078 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="34bb7979-7eb6-4b2f-a7d2-90acc05254f4" containerName="cleanup" Oct 30 18:15:23.799609 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.799083 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="34bb7979-7eb6-4b2f-a7d2-90acc05254f4" containerName="cleanup" Oct 30 18:15:23.799609 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.799090 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a3dd61-3a4f-40b9-8c5f-0896c37c5029" containerName="cleanup" Oct 30 18:15:23.799609 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.799094 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="3060b110-ab37-472b-b82a-c360d5bbd9a8" containerName="cleanup" Oct 30 18:15:23.802787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.802764 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.838125 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.838092 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj"] Oct 30 18:15:23.847725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.847693 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.847896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.847803 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.847896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.847829 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.848097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.847926 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.848097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.848013 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.848097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.848079 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zktb7\" (UniqueName: \"kubernetes.io/projected/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-kube-api-access-zktb7\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.848263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.848215 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.848315 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.848279 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.848371 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.848316 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.949827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.949748 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.949827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.949787 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.949827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.949820 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.950158 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.949879 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.950158 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.949916 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zktb7\" (UniqueName: \"kubernetes.io/projected/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-kube-api-access-zktb7\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.950158 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.950033 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.950158 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.950127 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.950158 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.950153 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.950393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.950219 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.950393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.950275 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-workload-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.950494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.950415 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-workload-certs\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.950697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.950670 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istiod-ca-cert\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.950778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.950744 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-data\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.950838 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.950790 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-credential-socket\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.952622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.952593 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-envoy\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.952949 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.952931 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-podinfo\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.969270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.969245 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zktb7\" (UniqueName: \"kubernetes.io/projected/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-kube-api-access-zktb7\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:23.969407 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:23.969293 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-token\") pod \"e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:24.125237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:24.125139 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:24.224071 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:24.223840 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" Oct 30 18:15:24.224071 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:24.223877 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb" event={"ID":"3060b110-ab37-472b-b82a-c360d5bbd9a8","Type":"ContainerDied","Data":"7b4212158b5c06cf7909493c242aeb3bed4fed15c842d0ab4f8d77777f1c3972"} Oct 30 18:15:24.224071 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:24.223951 5232 scope.go:122] "RemoveContainer" containerID="1a55881cf040815c190142b13f7ef72df32e2cc40fcb81cf9d82ab8d021813d8" Oct 30 18:15:24.280792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:24.280746 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb"] Oct 30 18:15:24.284710 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:24.284687 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889735-2mqhb"] Oct 30 18:15:24.487262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:24.487216 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:15:24.487492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:24.487462 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:15:24.487492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:24.487543 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:15:24.488491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:24.488468 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj"] Oct 30 18:15:24.805020 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:24.804982 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3060b110-ab37-472b-b82a-c360d5bbd9a8" path="/var/lib/kubelet/pods/3060b110-ab37-472b-b82a-c360d5bbd9a8/volumes" Oct 30 18:15:24.805420 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:24.805407 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44a3dd61-3a4f-40b9-8c5f-0896c37c5029" path="/var/lib/kubelet/pods/44a3dd61-3a4f-40b9-8c5f-0896c37c5029/volumes" Oct 30 18:15:25.246133 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:25.246096 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" event={"ID":"03bc2ff1-ddb6-41cf-8238-4df762a46c0e","Type":"ContainerStarted","Data":"8ef32012e3b73a4bf374336fbd318d9733b1ab44fcc489310ff6d813e835c031"} Oct 30 18:15:25.246542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:25.246155 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" event={"ID":"03bc2ff1-ddb6-41cf-8238-4df762a46c0e","Type":"ContainerStarted","Data":"ac440a882a062dd282671299bfd7e9b6d670729b861394be78f1422e2f4b5ffa"} Oct 30 18:15:26.134771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:26.134731 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:26.136865 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:26.136837 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:15:26.136999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:26.136897 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" podUID="03bc2ff1-ddb6-41cf-8238-4df762a46c0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:15:27.055103 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:15:27.055058 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:15:27.129533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:27.128943 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:15:27.129533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:27.129042 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" podUID="03bc2ff1-ddb6-41cf-8238-4df762a46c0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:15:28.134297 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:28.129927 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:15:28.134297 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:28.130005 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" podUID="03bc2ff1-ddb6-41cf-8238-4df762a46c0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:15:28.333900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:28.333870 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:28.348694 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:28.348666 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:28.392380 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:28.392280 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" podStartSLOduration=5.392264819 podStartE2EDuration="5.392264819s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:25.331848527 +0000 UTC m=+824.648268848" watchObservedRunningTime="2026-10-30 18:15:28.392264819 +0000 UTC m=+827.708685141" Oct 30 18:15:29.055283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.055250 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g"] Oct 30 18:15:29.056483 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.056459 5232 cpu_manager.go:414] "containerMap: removing container" podUID="ab7c2d8a-88e9-451c-938a-56a685d17a26" containerName="cleanup" Oct 30 18:15:29.056692 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.056674 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab7c2d8a-88e9-451c-938a-56a685d17a26" containerName="cleanup" Oct 30 18:15:29.056823 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.056802 5232 cpu_manager.go:414] "containerMap: removing container" podUID="44a3dd61-3a4f-40b9-8c5f-0896c37c5029" containerName="cleanup" Oct 30 18:15:29.056910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.056899 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44a3dd61-3a4f-40b9-8c5f-0896c37c5029" containerName="cleanup" Oct 30 18:15:29.057083 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.057070 5232 cpu_manager.go:414] "containerMap: removing container" podUID="367e7ada-dcd6-47fc-9117-76a1d38c84ed" containerName="cleanup" Oct 30 18:15:29.057201 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.057189 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="367e7ada-dcd6-47fc-9117-76a1d38c84ed" containerName="cleanup" Oct 30 18:15:29.057298 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.057288 5232 cpu_manager.go:414] "containerMap: removing container" podUID="3060b110-ab37-472b-b82a-c360d5bbd9a8" containerName="cleanup" Oct 30 18:15:29.074896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.074638 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3060b110-ab37-472b-b82a-c360d5bbd9a8" containerName="cleanup" Oct 30 18:15:29.075038 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.074969 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="44a3dd61-3a4f-40b9-8c5f-0896c37c5029" containerName="cleanup" Oct 30 18:15:29.075038 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.074990 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="367e7ada-dcd6-47fc-9117-76a1d38c84ed" containerName="cleanup" Oct 30 18:15:29.084376 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.083815 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g" Oct 30 18:15:29.084376 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.084257 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g"] Oct 30 18:15:29.098560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.098534 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:15:29.135544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.134652 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:15:29.135544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.134705 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" podUID="03bc2ff1-ddb6-41cf-8238-4df762a46c0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:15:29.136068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.135839 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:15:29.173683 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.173646 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c339c6e-e93b-42c1-860d-b83a3444a58b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1c4c4d6-f625g\" (UID: \"1c339c6e-e93b-42c1-860d-b83a3444a58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g" Oct 30 18:15:29.174244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.173773 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r769z\" (UniqueName: \"kubernetes.io/projected/1c339c6e-e93b-42c1-860d-b83a3444a58b-kube-api-access-r769z\") pod \"maas-api-revoke-sub-a1c4c4d6-f625g\" (UID: \"1c339c6e-e93b-42c1-860d-b83a3444a58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g" Oct 30 18:15:29.174244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.173888 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c339c6e-e93b-42c1-860d-b83a3444a58b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1c4c4d6-f625g\" (UID: \"1c339c6e-e93b-42c1-860d-b83a3444a58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g" Oct 30 18:15:29.274979 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.274951 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c339c6e-e93b-42c1-860d-b83a3444a58b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1c4c4d6-f625g\" (UID: \"1c339c6e-e93b-42c1-860d-b83a3444a58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g" Oct 30 18:15:29.275193 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.275030 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r769z\" (UniqueName: \"kubernetes.io/projected/1c339c6e-e93b-42c1-860d-b83a3444a58b-kube-api-access-r769z\") pod \"maas-api-revoke-sub-a1c4c4d6-f625g\" (UID: \"1c339c6e-e93b-42c1-860d-b83a3444a58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g" Oct 30 18:15:29.275193 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.275100 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c339c6e-e93b-42c1-860d-b83a3444a58b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1c4c4d6-f625g\" (UID: \"1c339c6e-e93b-42c1-860d-b83a3444a58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g" Oct 30 18:15:29.275761 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.275737 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c339c6e-e93b-42c1-860d-b83a3444a58b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a1c4c4d6-f625g\" (UID: \"1c339c6e-e93b-42c1-860d-b83a3444a58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g" Oct 30 18:15:29.278071 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.278046 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c339c6e-e93b-42c1-860d-b83a3444a58b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a1c4c4d6-f625g\" (UID: \"1c339c6e-e93b-42c1-860d-b83a3444a58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g" Oct 30 18:15:29.288119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.288093 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:15:29.295815 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.295791 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r769z\" (UniqueName: \"kubernetes.io/projected/1c339c6e-e93b-42c1-860d-b83a3444a58b-kube-api-access-r769z\") pod \"maas-api-revoke-sub-a1c4c4d6-f625g\" (UID: \"1c339c6e-e93b-42c1-860d-b83a3444a58b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g" Oct 30 18:15:29.399561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.399454 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g" Oct 30 18:15:29.749252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:29.749197 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g"] Oct 30 18:15:30.134043 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:30.134010 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:15:30.135540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:30.134296 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" podUID="03bc2ff1-ddb6-41cf-8238-4df762a46c0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:15:30.287561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:30.285051 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g" event={"ID":"1c339c6e-e93b-42c1-860d-b83a3444a58b","Type":"ContainerStarted","Data":"0203b6e989f1d905c1d841e560c13339287753be4d207f36cdc050c928ff2dbf"} Oct 30 18:15:30.287561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:30.285099 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g" event={"ID":"1c339c6e-e93b-42c1-860d-b83a3444a58b","Type":"ContainerStarted","Data":"9e4d6bfff2a7153268520dcbb646eebe294a9b3938bae62af96d773db8587347"} Oct 30 18:15:30.338997 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:30.338923 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g" podStartSLOduration=2.338909281 podStartE2EDuration="2.338909281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:30.332910389 +0000 UTC m=+829.649330712" watchObservedRunningTime="2026-10-30 18:15:30.338909281 +0000 UTC m=+829.655329656" Oct 30 18:15:31.126194 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:31.126150 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" start-of-body= Oct 30 18:15:31.126378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:31.126203 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" podUID="03bc2ff1-ddb6-41cf-8238-4df762a46c0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": dial tcp 10.133.0.91:15021: connect: connection refused" Oct 30 18:15:31.295235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:31.295195 5232 generic.go:356] "Generic (PLEG): container finished" podID="1c339c6e-e93b-42c1-860d-b83a3444a58b" containerID="0203b6e989f1d905c1d841e560c13339287753be4d207f36cdc050c928ff2dbf" exitCode=0 Oct 30 18:15:31.295788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:31.295282 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g" event={"ID":"1c339c6e-e93b-42c1-860d-b83a3444a58b","Type":"ContainerDied","Data":"0203b6e989f1d905c1d841e560c13339287753be4d207f36cdc050c928ff2dbf"} Oct 30 18:15:32.130067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:32.130040 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:32.130463 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:32.130445 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:32.131116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:32.131102 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:15:32.475491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:32.475467 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g" Oct 30 18:15:32.624877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:32.624846 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c339c6e-e93b-42c1-860d-b83a3444a58b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c339c6e-e93b-42c1-860d-b83a3444a58b-maas-api-openshift-ca\") pod \"1c339c6e-e93b-42c1-860d-b83a3444a58b\" (UID: \"1c339c6e-e93b-42c1-860d-b83a3444a58b\") " Oct 30 18:15:32.625058 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:32.625046 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c339c6e-e93b-42c1-860d-b83a3444a58b-kube-api-access-r769z\" (UniqueName: \"kubernetes.io/projected/1c339c6e-e93b-42c1-860d-b83a3444a58b-kube-api-access-r769z\") pod \"1c339c6e-e93b-42c1-860d-b83a3444a58b\" (UID: \"1c339c6e-e93b-42c1-860d-b83a3444a58b\") " Oct 30 18:15:32.625098 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:32.625069 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c339c6e-e93b-42c1-860d-b83a3444a58b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c339c6e-e93b-42c1-860d-b83a3444a58b-maas-api-xks-ca\") pod \"1c339c6e-e93b-42c1-860d-b83a3444a58b\" (UID: \"1c339c6e-e93b-42c1-860d-b83a3444a58b\") " Oct 30 18:15:32.625195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:32.625166 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c339c6e-e93b-42c1-860d-b83a3444a58b-maas-api-openshift-ca" pod "1c339c6e-e93b-42c1-860d-b83a3444a58b" (UID: "1c339c6e-e93b-42c1-860d-b83a3444a58b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:15:32.625460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:32.625447 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c339c6e-e93b-42c1-860d-b83a3444a58b-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:15:32.627372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:32.627346 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c339c6e-e93b-42c1-860d-b83a3444a58b-maas-api-xks-ca" pod "1c339c6e-e93b-42c1-860d-b83a3444a58b" (UID: "1c339c6e-e93b-42c1-860d-b83a3444a58b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:15:32.627457 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:32.627376 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c339c6e-e93b-42c1-860d-b83a3444a58b-kube-api-access-r769z" pod "1c339c6e-e93b-42c1-860d-b83a3444a58b" (UID: "1c339c6e-e93b-42c1-860d-b83a3444a58b"). InnerVolumeSpecName "kube-api-access-r769z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:15:32.726623 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:32.726543 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r769z\" (UniqueName: \"kubernetes.io/projected/1c339c6e-e93b-42c1-860d-b83a3444a58b-kube-api-access-r769z\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:15:32.726623 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:32.726571 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c339c6e-e93b-42c1-860d-b83a3444a58b-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:15:33.311259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:33.311220 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g" event={"ID":"1c339c6e-e93b-42c1-860d-b83a3444a58b","Type":"ContainerDied","Data":"9e4d6bfff2a7153268520dcbb646eebe294a9b3938bae62af96d773db8587347"} Oct 30 18:15:33.311545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:33.311284 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9e4d6bfff2a7153268520dcbb646eebe294a9b3938bae62af96d773db8587347" Oct 30 18:15:33.311545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:33.311235 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g" Oct 30 18:15:38.043571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.043538 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz"] Oct 30 18:15:38.044460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.044429 5232 cpu_manager.go:414] "containerMap: removing container" podUID="1c339c6e-e93b-42c1-860d-b83a3444a58b" containerName="revoke-keys" Oct 30 18:15:38.044460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.044455 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c339c6e-e93b-42c1-860d-b83a3444a58b" containerName="revoke-keys" Oct 30 18:15:38.044705 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.044689 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c339c6e-e93b-42c1-860d-b83a3444a58b" containerName="revoke-keys" Oct 30 18:15:38.070234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.070199 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz"] Oct 30 18:15:38.070416 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.070335 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.097477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.097454 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-8b6cf1\"/\"distinct-2-w3-8b6cf1-kserve-self-signed-certs\"" Oct 30 18:15:38.214195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.214156 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70bbbb7-6e09-46cf-9119-868fc52bb412-tls-certs\") pod \"distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.214378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.214210 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grnvd\" (UniqueName: \"kubernetes.io/projected/e70bbbb7-6e09-46cf-9119-868fc52bb412-kube-api-access-grnvd\") pod \"distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.214378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.214343 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-tmp-dir\") pod \"distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.214480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.214399 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-home\") pod \"distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.214538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.214492 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-dshm\") pod \"distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.214647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.214630 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-model-cache\") pod \"distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.327589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.318090 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-home\") pod \"distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.327589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.318149 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-dshm\") pod \"distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.327589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.318321 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-model-cache\") pod \"distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.327589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.318407 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70bbbb7-6e09-46cf-9119-868fc52bb412-tls-certs\") pod \"distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.327589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.318448 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grnvd\" (UniqueName: \"kubernetes.io/projected/e70bbbb7-6e09-46cf-9119-868fc52bb412-kube-api-access-grnvd\") pod \"distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.327589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.318567 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-tmp-dir\") pod \"distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.327589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.319910 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-tmp-dir\") pod \"distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.327589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.320166 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-model-cache\") pod \"distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.327589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.320556 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-home\") pod \"distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.342837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.342803 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-dshm\") pod \"distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.349019 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.348849 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grnvd\" (UniqueName: \"kubernetes.io/projected/e70bbbb7-6e09-46cf-9119-868fc52bb412-kube-api-access-grnvd\") pod \"distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.369532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.367583 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70bbbb7-6e09-46cf-9119-868fc52bb412-tls-certs\") pod \"distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.386410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.386374 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:38.745772 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:38.745744 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz"] Oct 30 18:15:38.746824 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:15:38.746780 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode70bbbb7_6e09_46cf_9119_868fc52bb412.slice/crio-6c51fdf7e6597f58cc63dec3702015724c25aa7623200b3373dd2eb37329c6fd WatchSource:0}: Error finding container 6c51fdf7e6597f58cc63dec3702015724c25aa7623200b3373dd2eb37329c6fd: Status 404 returned error can't find the container with id 6c51fdf7e6597f58cc63dec3702015724c25aa7623200b3373dd2eb37329c6fd Oct 30 18:15:39.455466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:39.455422 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" event={"ID":"e70bbbb7-6e09-46cf-9119-868fc52bb412","Type":"ContainerStarted","Data":"8aeabf78f92ea63322770ddcba8644e2bc0939d47e9ce78e94c579eae24513ec"} Oct 30 18:15:39.455917 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:39.455480 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" event={"ID":"e70bbbb7-6e09-46cf-9119-868fc52bb412","Type":"ContainerStarted","Data":"6c51fdf7e6597f58cc63dec3702015724c25aa7623200b3373dd2eb37329c6fd"} Oct 30 18:15:39.522532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:39.519051 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podStartSLOduration=1.519030457 podStartE2EDuration="1.519030457s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:15:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:15:39.517899383 +0000 UTC m=+838.834319704" watchObservedRunningTime="2026-10-30 18:15:39.519030457 +0000 UTC m=+838.835450779" Oct 30 18:15:40.388536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:40.387853 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:41.619707 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:41.619660 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:41.828936 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:41.824614 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:42.177756 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:42.177633 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:42.187120 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:42.187070 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:15:42.240670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:42.240622 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:15:42.305653 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:42.305604 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:42.306338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:42.305902 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:15:42.495569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:42.494620 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:42.671545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:42.669681 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:42.671545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:42.670354 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:15:42.831121 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:42.831008 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:15:42.955539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:42.955128 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:15:43.162561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:43.161791 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:15:43.190900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:43.189817 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:43.194560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:43.191499 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:15:43.195219 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:43.195190 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:43.197021 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:43.196998 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:15:43.217118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:43.217093 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:15:43.243444 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:43.243402 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:15:43.246297 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:43.244284 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:15:43.289089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:43.289059 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:15:43.290982 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:43.290674 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:15:43.307549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:43.307032 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:15:48.387342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:48.387308 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:48.400345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:48.400314 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:48.851182 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:15:48.851151 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:15:57.033798 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:15:57.033749 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:16:01.371101 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:01.371067 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r"] Oct 30 18:16:01.375571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:01.375551 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r" Oct 30 18:16:01.394856 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:01.394823 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r"] Oct 30 18:16:01.401146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:01.401118 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:16:01.401146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:01.401135 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:16:01.442079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:01.442047 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zvxg\" (UniqueName: \"kubernetes.io/projected/8ea92922-8471-45cf-bcf8-f70b41ead047-kube-api-access-9zvxg\") pod \"maas-api-revoke-sub-25a099a4-v5p7r\" (UID: \"8ea92922-8471-45cf-bcf8-f70b41ead047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r" Oct 30 18:16:01.442079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:01.442087 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea92922-8471-45cf-bcf8-f70b41ead047-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25a099a4-v5p7r\" (UID: \"8ea92922-8471-45cf-bcf8-f70b41ead047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r" Oct 30 18:16:01.442334 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:01.442118 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea92922-8471-45cf-bcf8-f70b41ead047-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25a099a4-v5p7r\" (UID: \"8ea92922-8471-45cf-bcf8-f70b41ead047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r" Oct 30 18:16:01.543439 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:01.543406 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea92922-8471-45cf-bcf8-f70b41ead047-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25a099a4-v5p7r\" (UID: \"8ea92922-8471-45cf-bcf8-f70b41ead047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r" Oct 30 18:16:01.543757 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:01.543739 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zvxg\" (UniqueName: \"kubernetes.io/projected/8ea92922-8471-45cf-bcf8-f70b41ead047-kube-api-access-9zvxg\") pod \"maas-api-revoke-sub-25a099a4-v5p7r\" (UID: \"8ea92922-8471-45cf-bcf8-f70b41ead047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r" Oct 30 18:16:01.543842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:01.543778 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea92922-8471-45cf-bcf8-f70b41ead047-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25a099a4-v5p7r\" (UID: \"8ea92922-8471-45cf-bcf8-f70b41ead047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r" Oct 30 18:16:01.544104 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:01.544083 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea92922-8471-45cf-bcf8-f70b41ead047-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-25a099a4-v5p7r\" (UID: \"8ea92922-8471-45cf-bcf8-f70b41ead047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r" Oct 30 18:16:01.546242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:01.546221 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea92922-8471-45cf-bcf8-f70b41ead047-maas-api-xks-ca\") pod \"maas-api-revoke-sub-25a099a4-v5p7r\" (UID: \"8ea92922-8471-45cf-bcf8-f70b41ead047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r" Oct 30 18:16:01.562784 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:01.562706 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zvxg\" (UniqueName: \"kubernetes.io/projected/8ea92922-8471-45cf-bcf8-f70b41ead047-kube-api-access-9zvxg\") pod \"maas-api-revoke-sub-25a099a4-v5p7r\" (UID: \"8ea92922-8471-45cf-bcf8-f70b41ead047\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r" Oct 30 18:16:01.691810 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:01.691723 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r" Oct 30 18:16:01.990301 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:01.990269 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r"] Oct 30 18:16:01.991334 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:16:01.991309 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ea92922_8471_45cf_bcf8_f70b41ead047.slice/crio-cf7b1febcaef4fa34849de58dd3e84c8ffc047841d69152362b87a5329ce0cdb WatchSource:0}: Error finding container cf7b1febcaef4fa34849de58dd3e84c8ffc047841d69152362b87a5329ce0cdb: Status 404 returned error can't find the container with id cf7b1febcaef4fa34849de58dd3e84c8ffc047841d69152362b87a5329ce0cdb Oct 30 18:16:02.966491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:02.966450 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r" event={"ID":"8ea92922-8471-45cf-bcf8-f70b41ead047","Type":"ContainerStarted","Data":"ff125f1132d1f0a41111489c74387db5b06d193bd3017cf63d81bc245aec9185"} Oct 30 18:16:02.966954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:02.966522 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r" event={"ID":"8ea92922-8471-45cf-bcf8-f70b41ead047","Type":"ContainerStarted","Data":"cf7b1febcaef4fa34849de58dd3e84c8ffc047841d69152362b87a5329ce0cdb"} Oct 30 18:16:03.005763 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:03.005717 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r" podStartSLOduration=2.005703075 podStartE2EDuration="2.005703075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:03.003445243 +0000 UTC m=+862.319865565" watchObservedRunningTime="2026-10-30 18:16:03.005703075 +0000 UTC m=+862.322123418" Oct 30 18:16:03.976342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:03.976305 5232 generic.go:356] "Generic (PLEG): container finished" podID="8ea92922-8471-45cf-bcf8-f70b41ead047" containerID="ff125f1132d1f0a41111489c74387db5b06d193bd3017cf63d81bc245aec9185" exitCode=0 Oct 30 18:16:03.976800 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:03.976393 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r" event={"ID":"8ea92922-8471-45cf-bcf8-f70b41ead047","Type":"ContainerDied","Data":"ff125f1132d1f0a41111489c74387db5b06d193bd3017cf63d81bc245aec9185"} Oct 30 18:16:05.163218 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:05.163193 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r" Oct 30 18:16:05.298870 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:05.298841 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ea92922-8471-45cf-bcf8-f70b41ead047-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea92922-8471-45cf-bcf8-f70b41ead047-maas-api-xks-ca\") pod \"8ea92922-8471-45cf-bcf8-f70b41ead047\" (UID: \"8ea92922-8471-45cf-bcf8-f70b41ead047\") " Oct 30 18:16:05.299070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:05.299003 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ea92922-8471-45cf-bcf8-f70b41ead047-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea92922-8471-45cf-bcf8-f70b41ead047-maas-api-openshift-ca\") pod \"8ea92922-8471-45cf-bcf8-f70b41ead047\" (UID: \"8ea92922-8471-45cf-bcf8-f70b41ead047\") " Oct 30 18:16:05.299169 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:05.299138 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ea92922-8471-45cf-bcf8-f70b41ead047-kube-api-access-9zvxg\" (UniqueName: \"kubernetes.io/projected/8ea92922-8471-45cf-bcf8-f70b41ead047-kube-api-access-9zvxg\") pod \"8ea92922-8471-45cf-bcf8-f70b41ead047\" (UID: \"8ea92922-8471-45cf-bcf8-f70b41ead047\") " Oct 30 18:16:05.299405 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:05.299377 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ea92922-8471-45cf-bcf8-f70b41ead047-maas-api-openshift-ca" pod "8ea92922-8471-45cf-bcf8-f70b41ead047" (UID: "8ea92922-8471-45cf-bcf8-f70b41ead047"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:05.299787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:05.299706 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8ea92922-8471-45cf-bcf8-f70b41ead047-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:05.301347 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:05.301261 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ea92922-8471-45cf-bcf8-f70b41ead047-maas-api-xks-ca" pod "8ea92922-8471-45cf-bcf8-f70b41ead047" (UID: "8ea92922-8471-45cf-bcf8-f70b41ead047"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:05.301347 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:05.301338 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ea92922-8471-45cf-bcf8-f70b41ead047-kube-api-access-9zvxg" pod "8ea92922-8471-45cf-bcf8-f70b41ead047" (UID: "8ea92922-8471-45cf-bcf8-f70b41ead047"). InnerVolumeSpecName "kube-api-access-9zvxg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:05.401549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:05.401500 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zvxg\" (UniqueName: \"kubernetes.io/projected/8ea92922-8471-45cf-bcf8-f70b41ead047-kube-api-access-9zvxg\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:05.401549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:05.401546 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8ea92922-8471-45cf-bcf8-f70b41ead047-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:05.990824 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:05.990791 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r" Oct 30 18:16:05.990824 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:05.990808 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r" event={"ID":"8ea92922-8471-45cf-bcf8-f70b41ead047","Type":"ContainerDied","Data":"cf7b1febcaef4fa34849de58dd3e84c8ffc047841d69152362b87a5329ce0cdb"} Oct 30 18:16:05.991028 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:05.990857 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf7b1febcaef4fa34849de58dd3e84c8ffc047841d69152362b87a5329ce0cdb" Oct 30 18:16:06.837625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:06.837594 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2"] Oct 30 18:16:06.838256 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:06.838238 5232 cpu_manager.go:414] "containerMap: removing container" podUID="8ea92922-8471-45cf-bcf8-f70b41ead047" containerName="revoke-keys" Oct 30 18:16:06.838256 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:06.838257 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ea92922-8471-45cf-bcf8-f70b41ead047" containerName="revoke-keys" Oct 30 18:16:06.838386 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:06.838374 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ea92922-8471-45cf-bcf8-f70b41ead047" containerName="revoke-keys" Oct 30 18:16:06.842062 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:06.842044 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:06.868589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:06.868559 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-fde2c5\"/\"distinct-2-w2-fde2c5-kserve-self-signed-certs\"" Oct 30 18:16:06.869726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:06.869699 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2"] Oct 30 18:16:06.930126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:06.930086 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-home\") pod \"distinct-2-w2-fde2c5-kserve-64b79db897-szqf2\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:06.930327 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:06.930158 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-model-cache\") pod \"distinct-2-w2-fde2c5-kserve-64b79db897-szqf2\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:06.930327 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:06.930270 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73249ea3-3ad3-489d-ac99-b1145098e719-tls-certs\") pod \"distinct-2-w2-fde2c5-kserve-64b79db897-szqf2\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:06.930418 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:06.930322 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbdk5\" (UniqueName: \"kubernetes.io/projected/73249ea3-3ad3-489d-ac99-b1145098e719-kube-api-access-tbdk5\") pod \"distinct-2-w2-fde2c5-kserve-64b79db897-szqf2\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:06.930596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:06.930561 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-tmp-dir\") pod \"distinct-2-w2-fde2c5-kserve-64b79db897-szqf2\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:06.930729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:06.930706 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-dshm\") pod \"distinct-2-w2-fde2c5-kserve-64b79db897-szqf2\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:07.031788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:07.031758 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-home\") pod \"distinct-2-w2-fde2c5-kserve-64b79db897-szqf2\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:07.031966 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:07.031821 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-model-cache\") pod \"distinct-2-w2-fde2c5-kserve-64b79db897-szqf2\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:07.031966 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:07.031848 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73249ea3-3ad3-489d-ac99-b1145098e719-tls-certs\") pod \"distinct-2-w2-fde2c5-kserve-64b79db897-szqf2\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:07.031966 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:07.031872 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbdk5\" (UniqueName: \"kubernetes.io/projected/73249ea3-3ad3-489d-ac99-b1145098e719-kube-api-access-tbdk5\") pod \"distinct-2-w2-fde2c5-kserve-64b79db897-szqf2\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:07.032089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:07.031994 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-tmp-dir\") pod \"distinct-2-w2-fde2c5-kserve-64b79db897-szqf2\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:07.032089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:07.032062 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-dshm\") pod \"distinct-2-w2-fde2c5-kserve-64b79db897-szqf2\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:07.032227 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:07.032161 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-home\") pod \"distinct-2-w2-fde2c5-kserve-64b79db897-szqf2\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:07.032276 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:07.032219 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-model-cache\") pod \"distinct-2-w2-fde2c5-kserve-64b79db897-szqf2\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:07.032431 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:07.032412 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-tmp-dir\") pod \"distinct-2-w2-fde2c5-kserve-64b79db897-szqf2\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:07.034381 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:07.034356 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-dshm\") pod \"distinct-2-w2-fde2c5-kserve-64b79db897-szqf2\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:07.034837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:07.034815 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73249ea3-3ad3-489d-ac99-b1145098e719-tls-certs\") pod \"distinct-2-w2-fde2c5-kserve-64b79db897-szqf2\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:07.053196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:07.053168 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbdk5\" (UniqueName: \"kubernetes.io/projected/73249ea3-3ad3-489d-ac99-b1145098e719-kube-api-access-tbdk5\") pod \"distinct-2-w2-fde2c5-kserve-64b79db897-szqf2\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:07.181577 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:07.178794 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:07.826930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:07.825965 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2"] Oct 30 18:16:07.830683 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:16:07.829838 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73249ea3_3ad3_489d_ac99_b1145098e719.slice/crio-65b73b1158764d0672754cc3f8ed8ee5750f7328b226d9cd700a9a669eabe7c3 WatchSource:0}: Error finding container 65b73b1158764d0672754cc3f8ed8ee5750f7328b226d9cd700a9a669eabe7c3: Status 404 returned error can't find the container with id 65b73b1158764d0672754cc3f8ed8ee5750f7328b226d9cd700a9a669eabe7c3 Oct 30 18:16:08.011108 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:08.011063 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" event={"ID":"73249ea3-3ad3-489d-ac99-b1145098e719","Type":"ContainerStarted","Data":"e96ffc5a00b0f002bc7762f5d7d7b1b1636ce2855821df81940ea3ba81ecb9a2"} Oct 30 18:16:08.011108 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:08.011124 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" event={"ID":"73249ea3-3ad3-489d-ac99-b1145098e719","Type":"ContainerStarted","Data":"65b73b1158764d0672754cc3f8ed8ee5750f7328b226d9cd700a9a669eabe7c3"} Oct 30 18:16:08.061586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:08.059330 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podStartSLOduration=2.059307777 podStartE2EDuration="2.059307777s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:08.051889639 +0000 UTC m=+867.368309961" watchObservedRunningTime="2026-10-30 18:16:08.059307777 +0000 UTC m=+867.375728098" Oct 30 18:16:08.182720 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:08.182608 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:09.179541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.169603 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:16:09.226538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.222666 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:09.289276 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.289233 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:09.291254 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.291223 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:09.292436 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.292412 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:16:09.323595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.323559 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:09.326315 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.326279 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:09.368224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.368192 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj"] Oct 30 18:16:09.373989 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.373964 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.385613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.385589 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt"] Oct 30 18:16:09.395777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.395753 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.407949 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.406184 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj"] Oct 30 18:16:09.408238 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.408214 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642"] Oct 30 18:16:09.437030 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.436965 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt"] Oct 30 18:16:09.437370 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.437353 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642"] Oct 30 18:16:09.437563 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.437294 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.561723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.561683 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.562006 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.561983 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.562164 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.562145 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.562308 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.562292 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.562430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.562413 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.562559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.562543 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b06967-baa7-4770-9632-f30f27f1dcae-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.562736 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.562716 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czr2r\" (UniqueName: \"kubernetes.io/projected/94b06967-baa7-4770-9632-f30f27f1dcae-kube-api-access-czr2r\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.562877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.562861 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.563010 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.562995 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.568737 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.568711 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b06967-baa7-4770-9632-f30f27f1dcae-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.569406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.569383 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7403bbb1-5c6e-431f-88a2-1005b4a22421-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.569596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.569575 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.569732 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.569715 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ebd246-b899-4a1d-9067-3993742fb028-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.569864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.569848 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b06967-baa7-4770-9632-f30f27f1dcae-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.570020 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.570002 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ebd246-b899-4a1d-9067-3993742fb028-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.570138 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.570123 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.570265 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.570250 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.570398 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.570383 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.570660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.570638 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.570832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.570812 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7wgpb\" (UniqueName: \"kubernetes.io/projected/7403bbb1-5c6e-431f-88a2-1005b4a22421-kube-api-access-7wgpb\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.571088 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.571068 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebd246-b899-4a1d-9067-3993742fb028-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.571239 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.571221 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcxg7\" (UniqueName: \"kubernetes.io/projected/f2ebd246-b899-4a1d-9067-3993742fb028-kube-api-access-gcxg7\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.571352 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.571335 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.571465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.571450 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.571708 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.571688 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.571861 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.571843 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.572111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.571979 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.618531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.611105 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:09.618531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.611166 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" podUID="f8c0e348-2b8d-48e2-9ec0-a21a83db391e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:09.678521 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.678300 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.678782 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.678761 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.678902 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.678888 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.679024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.679011 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.679155 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.679140 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.679269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.679255 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.679360 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.679348 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b06967-baa7-4770-9632-f30f27f1dcae-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.679521 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.679479 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czr2r\" (UniqueName: \"kubernetes.io/projected/94b06967-baa7-4770-9632-f30f27f1dcae-kube-api-access-czr2r\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.679652 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.679637 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.679767 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.679755 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.679868 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.679857 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b06967-baa7-4770-9632-f30f27f1dcae-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.679962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.679949 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7403bbb1-5c6e-431f-88a2-1005b4a22421-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.680054 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.680042 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.680156 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.680143 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ebd246-b899-4a1d-9067-3993742fb028-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.680254 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.680243 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b06967-baa7-4770-9632-f30f27f1dcae-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.680377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.680365 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ebd246-b899-4a1d-9067-3993742fb028-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.680471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.680459 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.680608 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.680592 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.680764 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.680747 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.680880 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.680865 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.681007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.680990 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7wgpb\" (UniqueName: \"kubernetes.io/projected/7403bbb1-5c6e-431f-88a2-1005b4a22421-kube-api-access-7wgpb\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.681121 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.681105 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebd246-b899-4a1d-9067-3993742fb028-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.681228 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.681214 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcxg7\" (UniqueName: \"kubernetes.io/projected/f2ebd246-b899-4a1d-9067-3993742fb028-kube-api-access-gcxg7\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.681334 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.681320 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.681444 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.681426 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.681671 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.681650 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.681827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.681811 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.682437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.682414 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.682928 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.682908 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.685577 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.683708 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.685709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.683841 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-workload-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.685806 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.683943 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-workload-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.685898 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.684286 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.685983 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.684729 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-istio-data\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.686312 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.686293 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-data\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.706011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.693426 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-credential-socket\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.706011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.693710 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-workload-certs\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.706011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.694325 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b06967-baa7-4770-9632-f30f27f1dcae-istiod-ca-cert\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.706011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.694984 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-workload-certs\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.706011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.695924 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7403bbb1-5c6e-431f-88a2-1005b4a22421-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.706011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.695996 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebd246-b899-4a1d-9067-3993742fb028-istiod-ca-cert\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.706011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.696222 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-credential-socket\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.706011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.696406 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw"] Oct 30 18:16:09.706011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.704365 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7wgpb\" (UniqueName: \"kubernetes.io/projected/7403bbb1-5c6e-431f-88a2-1005b4a22421-kube-api-access-7wgpb\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.706011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.705405 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.706011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.705988 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b06967-baa7-4770-9632-f30f27f1dcae-istio-podinfo\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.706802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.706062 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ebd246-b899-4a1d-9067-3993742fb028-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.706802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.706096 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ebd246-b899-4a1d-9067-3993742fb028-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.706802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.706498 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcxg7\" (UniqueName: \"kubernetes.io/projected/f2ebd246-b899-4a1d-9067-3993742fb028-kube-api-access-gcxg7\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.706802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.706779 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-envoy\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.707030 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.706802 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-istio-envoy\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.707632 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.707607 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-podinfo\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.707902 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.707883 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw"] Oct 30 18:16:09.708005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.707990 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw" Oct 30 18:16:09.708902 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.708876 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-token\") pod \"e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.715325 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.714042 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czr2r\" (UniqueName: \"kubernetes.io/projected/94b06967-baa7-4770-9632-f30f27f1dcae-kube-api-access-czr2r\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.715325 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.714909 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b06967-baa7-4770-9632-f30f27f1dcae-istio-token\") pod \"e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.718155 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.718132 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:09.727282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.723847 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:16:09.727282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.725028 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:16:09.735818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.735791 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:09.786607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.783366 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07959221-b081-4dec-ab32-2b3adbeee19b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e3b4f3-vcnfw\" (UID: \"07959221-b081-4dec-ab32-2b3adbeee19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw" Oct 30 18:16:09.786607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.783410 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07959221-b081-4dec-ab32-2b3adbeee19b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e3b4f3-vcnfw\" (UID: \"07959221-b081-4dec-ab32-2b3adbeee19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw" Oct 30 18:16:09.786607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.783707 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fljq8\" (UniqueName: \"kubernetes.io/projected/07959221-b081-4dec-ab32-2b3adbeee19b-kube-api-access-fljq8\") pod \"maas-api-revoke-sub-11e3b4f3-vcnfw\" (UID: \"07959221-b081-4dec-ab32-2b3adbeee19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw" Oct 30 18:16:09.831066 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.830882 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:09.885454 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.885415 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fljq8\" (UniqueName: \"kubernetes.io/projected/07959221-b081-4dec-ab32-2b3adbeee19b-kube-api-access-fljq8\") pod \"maas-api-revoke-sub-11e3b4f3-vcnfw\" (UID: \"07959221-b081-4dec-ab32-2b3adbeee19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw" Oct 30 18:16:09.885675 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.885659 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07959221-b081-4dec-ab32-2b3adbeee19b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e3b4f3-vcnfw\" (UID: \"07959221-b081-4dec-ab32-2b3adbeee19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw" Oct 30 18:16:09.886768 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.886743 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07959221-b081-4dec-ab32-2b3adbeee19b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e3b4f3-vcnfw\" (UID: \"07959221-b081-4dec-ab32-2b3adbeee19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw" Oct 30 18:16:09.887576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.885704 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07959221-b081-4dec-ab32-2b3adbeee19b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-11e3b4f3-vcnfw\" (UID: \"07959221-b081-4dec-ab32-2b3adbeee19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw" Oct 30 18:16:09.888785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.888761 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07959221-b081-4dec-ab32-2b3adbeee19b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-11e3b4f3-vcnfw\" (UID: \"07959221-b081-4dec-ab32-2b3adbeee19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw" Oct 30 18:16:09.909607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:09.906557 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fljq8\" (UniqueName: \"kubernetes.io/projected/07959221-b081-4dec-ab32-2b3adbeee19b-kube-api-access-fljq8\") pod \"maas-api-revoke-sub-11e3b4f3-vcnfw\" (UID: \"07959221-b081-4dec-ab32-2b3adbeee19b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw" Oct 30 18:16:10.107846 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.107791 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw" Oct 30 18:16:10.118868 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:16:10.118783 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7403bbb1_5c6e_431f_88a2_1005b4a22421.slice/crio-7d3fcc8b33b08512b456b78b237ccd0e673109ebb855f7a556f62fc9e077d120 WatchSource:0}: Error finding container 7d3fcc8b33b08512b456b78b237ccd0e673109ebb855f7a556f62fc9e077d120: Status 404 returned error can't find the container with id 7d3fcc8b33b08512b456b78b237ccd0e673109ebb855f7a556f62fc9e077d120 Oct 30 18:16:10.121559 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:16:10.121526 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod94b06967_baa7_4770_9632_f30f27f1dcae.slice/crio-db92124d02160507e9e75851a18e0fb12af7d9a8b01348712a06c6ca07f40ff0 WatchSource:0}: Error finding container db92124d02160507e9e75851a18e0fb12af7d9a8b01348712a06c6ca07f40ff0: Status 404 returned error can't find the container with id db92124d02160507e9e75851a18e0fb12af7d9a8b01348712a06c6ca07f40ff0 Oct 30 18:16:10.122831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.122794 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:16:10.122894 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.122874 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:16:10.122939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.122917 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:16:10.124568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.124493 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:16:10.124630 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.124615 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:16:10.124667 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.124656 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:16:10.125995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.125980 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj"] Oct 30 18:16:10.130572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.130552 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt"] Oct 30 18:16:10.161638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.161449 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642"] Oct 30 18:16:10.178997 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:16:10.178839 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2ebd246_b899_4a1d_9067_3993742fb028.slice/crio-deccf08beaa3e3870205db4707a6db43f33793f652559011c6a65b921ff9d320 WatchSource:0}: Error finding container deccf08beaa3e3870205db4707a6db43f33793f652559011c6a65b921ff9d320: Status 404 returned error can't find the container with id deccf08beaa3e3870205db4707a6db43f33793f652559011c6a65b921ff9d320 Oct 30 18:16:10.185718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.185661 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:16:10.186107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.185777 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:16:10.186107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.185822 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:16:10.371540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.371212 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" event={"ID":"94b06967-baa7-4770-9632-f30f27f1dcae","Type":"ContainerStarted","Data":"f0c3ce9cefec72aa003aad71bbf47d3f013da4f47c46676a85da9c7380cc124c"} Oct 30 18:16:10.371540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.371282 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" event={"ID":"94b06967-baa7-4770-9632-f30f27f1dcae","Type":"ContainerStarted","Data":"db92124d02160507e9e75851a18e0fb12af7d9a8b01348712a06c6ca07f40ff0"} Oct 30 18:16:10.401619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.401579 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" event={"ID":"f2ebd246-b899-4a1d-9067-3993742fb028","Type":"ContainerStarted","Data":"aced874e598d3925ad113b36279d79937cb7ebbc84ed299c2493a0ff8f63e602"} Oct 30 18:16:10.401944 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.401919 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" event={"ID":"f2ebd246-b899-4a1d-9067-3993742fb028","Type":"ContainerStarted","Data":"deccf08beaa3e3870205db4707a6db43f33793f652559011c6a65b921ff9d320"} Oct 30 18:16:10.408543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.405652 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" event={"ID":"7403bbb1-5c6e-431f-88a2-1005b4a22421","Type":"ContainerStarted","Data":"74774892dbfbb1968405fcfb47f5d0201b3a56715cedc289a60c85aeb68bb854"} Oct 30 18:16:10.408543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.405711 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" event={"ID":"7403bbb1-5c6e-431f-88a2-1005b4a22421","Type":"ContainerStarted","Data":"7d3fcc8b33b08512b456b78b237ccd0e673109ebb855f7a556f62fc9e077d120"} Oct 30 18:16:10.435273 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.435218 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" podStartSLOduration=2.435199849 podStartE2EDuration="2.435199849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:10.428352989 +0000 UTC m=+869.744773311" watchObservedRunningTime="2026-10-30 18:16:10.435199849 +0000 UTC m=+869.751620170" Oct 30 18:16:10.444663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.444639 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw"] Oct 30 18:16:10.447672 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:16:10.447642 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07959221_b081_4dec_ab32_2b3adbeee19b.slice/crio-55216914d58e9181604265d0332b98632e43f024fab97ed282d6fa0997d5807f WatchSource:0}: Error finding container 55216914d58e9181604265d0332b98632e43f024fab97ed282d6fa0997d5807f: Status 404 returned error can't find the container with id 55216914d58e9181604265d0332b98632e43f024fab97ed282d6fa0997d5807f Oct 30 18:16:10.475656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.475595 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" podStartSLOduration=2.475579104 podStartE2EDuration="2.475579104s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:10.469789889 +0000 UTC m=+869.786210211" watchObservedRunningTime="2026-10-30 18:16:10.475579104 +0000 UTC m=+869.791999427" Oct 30 18:16:10.515365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.515316 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" podStartSLOduration=2.515297597 podStartE2EDuration="2.515297597s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:10.512592189 +0000 UTC m=+869.829012511" watchObservedRunningTime="2026-10-30 18:16:10.515297597 +0000 UTC m=+869.831717921" Oct 30 18:16:10.719085 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.719056 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:10.736613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.736570 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:10.831756 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:10.831724 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:11.469533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:11.461102 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw" event={"ID":"07959221-b081-4dec-ab32-2b3adbeee19b","Type":"ContainerStarted","Data":"715bd0736dd3b9a0b96094fc2db7ad9f4a7712b3b4886310eb0e230445af6680"} Oct 30 18:16:11.469533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:11.461177 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw" event={"ID":"07959221-b081-4dec-ab32-2b3adbeee19b","Type":"ContainerStarted","Data":"55216914d58e9181604265d0332b98632e43f024fab97ed282d6fa0997d5807f"} Oct 30 18:16:11.503869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:11.503733 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw" podStartSLOduration=2.5037134610000003 podStartE2EDuration="2.503713461s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:11.503270298 +0000 UTC m=+870.819690620" watchObservedRunningTime="2026-10-30 18:16:11.503713461 +0000 UTC m=+870.820133782" Oct 30 18:16:11.737602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:11.726860 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:16:11.737602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:11.726963 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" podUID="94b06967-baa7-4770-9632-f30f27f1dcae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:16:11.745532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:11.740387 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:16:11.745532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:11.740525 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" podUID="7403bbb1-5c6e-431f-88a2-1005b4a22421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 18:16:11.838193 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:11.837223 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:16:11.838193 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:11.837293 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" podUID="f2ebd246-b899-4a1d-9067-3993742fb028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 18:16:12.272534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.271581 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92"] Oct 30 18:16:12.332567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.331796 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92" Oct 30 18:16:12.455549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.454574 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92"] Oct 30 18:16:12.502529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.486086 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-clzt7\" (UniqueName: \"kubernetes.io/projected/734d033e-b406-4d56-a09f-ac34c9bf0951-kube-api-access-clzt7\") pod \"maas-api-revoke-sub-58634cfb-d8s92\" (UID: \"734d033e-b406-4d56-a09f-ac34c9bf0951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92" Oct 30 18:16:12.502529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.486228 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734d033e-b406-4d56-a09f-ac34c9bf0951-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58634cfb-d8s92\" (UID: \"734d033e-b406-4d56-a09f-ac34c9bf0951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92" Oct 30 18:16:12.502529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.486396 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734d033e-b406-4d56-a09f-ac34c9bf0951-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58634cfb-d8s92\" (UID: \"734d033e-b406-4d56-a09f-ac34c9bf0951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92" Oct 30 18:16:12.590701 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.590583 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-clzt7\" (UniqueName: \"kubernetes.io/projected/734d033e-b406-4d56-a09f-ac34c9bf0951-kube-api-access-clzt7\") pod \"maas-api-revoke-sub-58634cfb-d8s92\" (UID: \"734d033e-b406-4d56-a09f-ac34c9bf0951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92" Oct 30 18:16:12.591411 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.591374 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734d033e-b406-4d56-a09f-ac34c9bf0951-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58634cfb-d8s92\" (UID: \"734d033e-b406-4d56-a09f-ac34c9bf0951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92" Oct 30 18:16:12.594846 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.594818 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734d033e-b406-4d56-a09f-ac34c9bf0951-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58634cfb-d8s92\" (UID: \"734d033e-b406-4d56-a09f-ac34c9bf0951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92" Oct 30 18:16:12.596252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.593781 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw" event={"ID":"07959221-b081-4dec-ab32-2b3adbeee19b","Type":"ContainerDied","Data":"715bd0736dd3b9a0b96094fc2db7ad9f4a7712b3b4886310eb0e230445af6680"} Oct 30 18:16:12.596422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.594622 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734d033e-b406-4d56-a09f-ac34c9bf0951-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-58634cfb-d8s92\" (UID: \"734d033e-b406-4d56-a09f-ac34c9bf0951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92" Oct 30 18:16:12.596542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.593756 5232 generic.go:356] "Generic (PLEG): container finished" podID="07959221-b081-4dec-ab32-2b3adbeee19b" containerID="715bd0736dd3b9a0b96094fc2db7ad9f4a7712b3b4886310eb0e230445af6680" exitCode=0 Oct 30 18:16:12.636530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.609459 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734d033e-b406-4d56-a09f-ac34c9bf0951-maas-api-xks-ca\") pod \"maas-api-revoke-sub-58634cfb-d8s92\" (UID: \"734d033e-b406-4d56-a09f-ac34c9bf0951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92" Oct 30 18:16:12.702392 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.698370 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-clzt7\" (UniqueName: \"kubernetes.io/projected/734d033e-b406-4d56-a09f-ac34c9bf0951-kube-api-access-clzt7\") pod \"maas-api-revoke-sub-58634cfb-d8s92\" (UID: \"734d033e-b406-4d56-a09f-ac34c9bf0951\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92" Oct 30 18:16:12.730710 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.720945 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:16:12.730710 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.721014 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" podUID="94b06967-baa7-4770-9632-f30f27f1dcae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:16:12.746536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.745609 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:16:12.746536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.745687 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" podUID="7403bbb1-5c6e-431f-88a2-1005b4a22421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 18:16:12.837641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.835212 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:16:12.837641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.835275 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" podUID="f2ebd246-b899-4a1d-9067-3993742fb028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 18:16:12.890609 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:12.883759 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92" Oct 30 18:16:13.755540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:13.754998 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:16:13.755540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:13.755062 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" podUID="94b06967-baa7-4770-9632-f30f27f1dcae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:16:13.793563 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:13.759579 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:16:13.793563 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:13.759639 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" podUID="7403bbb1-5c6e-431f-88a2-1005b4a22421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 18:16:13.813824 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:13.809287 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92"] Oct 30 18:16:13.832399 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:13.831962 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:16:13.832399 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:13.832017 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" podUID="f2ebd246-b899-4a1d-9067-3993742fb028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 18:16:13.847224 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:16:13.847173 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod734d033e_b406_4d56_a09f_ac34c9bf0951.slice/crio-e40e8e2e353bf08b4530603b470b655d5ba147c821e9e59668987ca0b7097543 WatchSource:0}: Error finding container e40e8e2e353bf08b4530603b470b655d5ba147c821e9e59668987ca0b7097543: Status 404 returned error can't find the container with id e40e8e2e353bf08b4530603b470b655d5ba147c821e9e59668987ca0b7097543 Oct 30 18:16:14.343556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:14.313700 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw" Oct 30 18:16:14.759539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:14.758782 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07959221-b081-4dec-ab32-2b3adbeee19b-kube-api-access-fljq8\" (UniqueName: \"kubernetes.io/projected/07959221-b081-4dec-ab32-2b3adbeee19b-kube-api-access-fljq8\") pod \"07959221-b081-4dec-ab32-2b3adbeee19b\" (UID: \"07959221-b081-4dec-ab32-2b3adbeee19b\") " Oct 30 18:16:14.759539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:14.758887 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07959221-b081-4dec-ab32-2b3adbeee19b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07959221-b081-4dec-ab32-2b3adbeee19b-maas-api-openshift-ca\") pod \"07959221-b081-4dec-ab32-2b3adbeee19b\" (UID: \"07959221-b081-4dec-ab32-2b3adbeee19b\") " Oct 30 18:16:14.759539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:14.758977 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07959221-b081-4dec-ab32-2b3adbeee19b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07959221-b081-4dec-ab32-2b3adbeee19b-maas-api-xks-ca\") pod \"07959221-b081-4dec-ab32-2b3adbeee19b\" (UID: \"07959221-b081-4dec-ab32-2b3adbeee19b\") " Oct 30 18:16:14.906431 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:14.906112 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07959221-b081-4dec-ab32-2b3adbeee19b-maas-api-openshift-ca" pod "07959221-b081-4dec-ab32-2b3adbeee19b" (UID: "07959221-b081-4dec-ab32-2b3adbeee19b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:14.908796 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:14.908770 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07959221-b081-4dec-ab32-2b3adbeee19b-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:14.966637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:14.931640 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:16:14.982256 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:14.982214 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" podUID="f2ebd246-b899-4a1d-9067-3993742fb028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 18:16:15.001125 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.001090 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:16:15.001381 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.001356 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" podUID="94b06967-baa7-4770-9632-f30f27f1dcae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:16:15.001991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.001961 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:16:15.004654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.004624 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" podUID="7403bbb1-5c6e-431f-88a2-1005b4a22421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 18:16:15.044080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.044045 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07959221-b081-4dec-ab32-2b3adbeee19b-kube-api-access-fljq8" pod "07959221-b081-4dec-ab32-2b3adbeee19b" (UID: "07959221-b081-4dec-ab32-2b3adbeee19b"). InnerVolumeSpecName "kube-api-access-fljq8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:15.067101 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.067065 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fljq8\" (UniqueName: \"kubernetes.io/projected/07959221-b081-4dec-ab32-2b3adbeee19b-kube-api-access-fljq8\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:15.076393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.076355 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07959221-b081-4dec-ab32-2b3adbeee19b-maas-api-xks-ca" pod "07959221-b081-4dec-ab32-2b3adbeee19b" (UID: "07959221-b081-4dec-ab32-2b3adbeee19b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:15.087702 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.087249 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr"] Oct 30 18:16:15.107460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.107432 5232 cpu_manager.go:414] "containerMap: removing container" podUID="07959221-b081-4dec-ab32-2b3adbeee19b" containerName="revoke-keys" Oct 30 18:16:15.107460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.107457 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07959221-b081-4dec-ab32-2b3adbeee19b" containerName="revoke-keys" Oct 30 18:16:15.107753 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.107735 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="07959221-b081-4dec-ab32-2b3adbeee19b" containerName="revoke-keys" Oct 30 18:16:15.134687 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.134650 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr"] Oct 30 18:16:15.135022 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.135002 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr" Oct 30 18:16:15.164116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.152788 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw" event={"ID":"07959221-b081-4dec-ab32-2b3adbeee19b","Type":"ContainerDied","Data":"55216914d58e9181604265d0332b98632e43f024fab97ed282d6fa0997d5807f"} Oct 30 18:16:15.164116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.152848 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="55216914d58e9181604265d0332b98632e43f024fab97ed282d6fa0997d5807f" Oct 30 18:16:15.164116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.152951 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw" Oct 30 18:16:15.170969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.169084 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07959221-b081-4dec-ab32-2b3adbeee19b-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:15.197592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.189447 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92" event={"ID":"734d033e-b406-4d56-a09f-ac34c9bf0951","Type":"ContainerStarted","Data":"e40e8e2e353bf08b4530603b470b655d5ba147c821e9e59668987ca0b7097543"} Oct 30 18:16:15.274533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.273806 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27mt7\" (UniqueName: \"kubernetes.io/projected/e8606381-4cb4-4096-9365-6e30d42a54ba-kube-api-access-27mt7\") pod \"maas-api-revoke-sub-d7cf2f23-86lnr\" (UID: \"e8606381-4cb4-4096-9365-6e30d42a54ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr" Oct 30 18:16:15.274533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.273930 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8606381-4cb4-4096-9365-6e30d42a54ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7cf2f23-86lnr\" (UID: \"e8606381-4cb4-4096-9365-6e30d42a54ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr" Oct 30 18:16:15.274533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.274161 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8606381-4cb4-4096-9365-6e30d42a54ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7cf2f23-86lnr\" (UID: \"e8606381-4cb4-4096-9365-6e30d42a54ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr" Oct 30 18:16:15.289430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.289378 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92" podStartSLOduration=3.2893647 podStartE2EDuration="3.2893647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:15.252974898 +0000 UTC m=+874.569395220" watchObservedRunningTime="2026-10-30 18:16:15.2893647 +0000 UTC m=+874.605785020" Oct 30 18:16:15.377538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.376537 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8606381-4cb4-4096-9365-6e30d42a54ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7cf2f23-86lnr\" (UID: \"e8606381-4cb4-4096-9365-6e30d42a54ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr" Oct 30 18:16:15.377538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.376775 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8606381-4cb4-4096-9365-6e30d42a54ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7cf2f23-86lnr\" (UID: \"e8606381-4cb4-4096-9365-6e30d42a54ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr" Oct 30 18:16:15.377538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.376898 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27mt7\" (UniqueName: \"kubernetes.io/projected/e8606381-4cb4-4096-9365-6e30d42a54ba-kube-api-access-27mt7\") pod \"maas-api-revoke-sub-d7cf2f23-86lnr\" (UID: \"e8606381-4cb4-4096-9365-6e30d42a54ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr" Oct 30 18:16:15.385589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.378142 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8606381-4cb4-4096-9365-6e30d42a54ba-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d7cf2f23-86lnr\" (UID: \"e8606381-4cb4-4096-9365-6e30d42a54ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr" Oct 30 18:16:15.392250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.392216 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8606381-4cb4-4096-9365-6e30d42a54ba-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d7cf2f23-86lnr\" (UID: \"e8606381-4cb4-4096-9365-6e30d42a54ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr" Oct 30 18:16:15.397385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.397350 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27mt7\" (UniqueName: \"kubernetes.io/projected/e8606381-4cb4-4096-9365-6e30d42a54ba-kube-api-access-27mt7\") pod \"maas-api-revoke-sub-d7cf2f23-86lnr\" (UID: \"e8606381-4cb4-4096-9365-6e30d42a54ba\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr" Oct 30 18:16:15.509471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.509435 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr" Oct 30 18:16:15.766551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.743936 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:16:15.766551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.743994 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" podUID="7403bbb1-5c6e-431f-88a2-1005b4a22421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 18:16:15.766551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.744656 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:16:15.766551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.744694 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" podUID="94b06967-baa7-4770-9632-f30f27f1dcae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:16:15.881733 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.876570 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:16:15.881733 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:15.876643 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" podUID="f2ebd246-b899-4a1d-9067-3993742fb028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 18:16:16.007862 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.007830 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr"] Oct 30 18:16:16.202141 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.202112 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr" event={"ID":"e8606381-4cb4-4096-9365-6e30d42a54ba","Type":"ContainerStarted","Data":"82e4baf2f287cc165d81ad01bc0d7606e71fb20ace7273b519a4a9f8d1f7306f"} Oct 30 18:16:16.204187 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.204164 5232 generic.go:356] "Generic (PLEG): container finished" podID="734d033e-b406-4d56-a09f-ac34c9bf0951" containerID="77e7d2c66f0777149ec0e5524b0b273e25786f37983ef9c888dc8b48c6141d53" exitCode=0 Oct 30 18:16:16.204326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.204304 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92" event={"ID":"734d033e-b406-4d56-a09f-ac34c9bf0951","Type":"ContainerDied","Data":"77e7d2c66f0777149ec0e5524b0b273e25786f37983ef9c888dc8b48c6141d53"} Oct 30 18:16:16.533732 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.517847 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb"] Oct 30 18:16:16.592535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.580942 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb" Oct 30 18:16:16.691542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.682089 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69dac4e0-rhmjb\" (UID: \"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb" Oct 30 18:16:16.691542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.682227 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69dac4e0-rhmjb\" (UID: \"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb" Oct 30 18:16:16.691542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.682380 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftskj\" (UniqueName: \"kubernetes.io/projected/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-kube-api-access-ftskj\") pod \"maas-api-revoke-sub-69dac4e0-rhmjb\" (UID: \"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb" Oct 30 18:16:16.721653 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.721619 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:16:16.721904 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.721874 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" podUID="94b06967-baa7-4770-9632-f30f27f1dcae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:16:16.739228 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.739198 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb"] Oct 30 18:16:16.764181 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.764147 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:16:16.764438 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.764411 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" podUID="7403bbb1-5c6e-431f-88a2-1005b4a22421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 18:16:16.802661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.797979 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69dac4e0-rhmjb\" (UID: \"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb" Oct 30 18:16:16.802661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.798084 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69dac4e0-rhmjb\" (UID: \"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb" Oct 30 18:16:16.802661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.798189 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftskj\" (UniqueName: \"kubernetes.io/projected/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-kube-api-access-ftskj\") pod \"maas-api-revoke-sub-69dac4e0-rhmjb\" (UID: \"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb" Oct 30 18:16:16.802661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.799128 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-69dac4e0-rhmjb\" (UID: \"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb" Oct 30 18:16:16.804257 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.804218 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-maas-api-xks-ca\") pod \"maas-api-revoke-sub-69dac4e0-rhmjb\" (UID: \"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb" Oct 30 18:16:16.843027 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.838765 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:16:16.843027 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.838819 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" podUID="f2ebd246-b899-4a1d-9067-3993742fb028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 18:16:16.892559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:16.886790 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftskj\" (UniqueName: \"kubernetes.io/projected/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-kube-api-access-ftskj\") pod \"maas-api-revoke-sub-69dac4e0-rhmjb\" (UID: \"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb" Oct 30 18:16:17.008011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.007976 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd"] Oct 30 18:16:17.080393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.080309 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.086356 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.086332 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd"] Oct 30 18:16:17.096781 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.096756 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb" Oct 30 18:16:17.101253 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.100930 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-fb2fa0\"/\"distinct-2-w1-fb2fa0-kserve-self-signed-certs\"" Oct 30 18:16:17.187828 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.187428 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:17.227539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.218206 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d74ee1cf-7d30-49b6-880d-a7368794823d-tls-certs\") pod \"distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.227539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.218263 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-dshm\") pod \"distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.227539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.218570 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-tmp-dir\") pod \"distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.227539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.218638 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-model-cache\") pod \"distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.227539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.218694 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wcw4z\" (UniqueName: \"kubernetes.io/projected/d74ee1cf-7d30-49b6-880d-a7368794823d-kube-api-access-wcw4z\") pod \"distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.304534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.295230 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-home\") pod \"distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.403583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.402897 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:17.409536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.406059 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-tmp-dir\") pod \"distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.409536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.406125 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-model-cache\") pod \"distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.409536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.406208 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wcw4z\" (UniqueName: \"kubernetes.io/projected/d74ee1cf-7d30-49b6-880d-a7368794823d-kube-api-access-wcw4z\") pod \"distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.409536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.406250 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-home\") pod \"distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.409536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.406368 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d74ee1cf-7d30-49b6-880d-a7368794823d-tls-certs\") pod \"distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.409536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.406414 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-dshm\") pod \"distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.409536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.407793 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-tmp-dir\") pod \"distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.409536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.408072 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-model-cache\") pod \"distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.409536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.408315 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-home\") pod \"distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.458573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.456150 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d74ee1cf-7d30-49b6-880d-a7368794823d-tls-certs\") pod \"distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.458573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.456383 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-dshm\") pod \"distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.484297 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.465592 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr" event={"ID":"e8606381-4cb4-4096-9365-6e30d42a54ba","Type":"ContainerStarted","Data":"6f1ab5924188988bfc26f587192f5cfd604155e5ae8e79c9d9f81aff76107724"} Oct 30 18:16:17.557185 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.553164 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wcw4z\" (UniqueName: \"kubernetes.io/projected/d74ee1cf-7d30-49b6-880d-a7368794823d-kube-api-access-wcw4z\") pod \"distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.565338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.563243 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:17.572910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.572841 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr" podStartSLOduration=3.5728170820000003 podStartE2EDuration="3.572817082s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:17.563615197 +0000 UTC m=+876.880035521" watchObservedRunningTime="2026-10-30 18:16:17.572817082 +0000 UTC m=+876.889237405" Oct 30 18:16:17.717669 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.717411 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:17.721197 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.721165 5232 patch_prober.go:29] interesting pod/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" start-of-body= Oct 30 18:16:17.721361 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.721218 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" podUID="94b06967-baa7-4770-9632-f30f27f1dcae" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.96:15021/healthz/ready\": dial tcp 10.133.0.96:15021: connect: connection refused" Oct 30 18:16:17.772264 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.772230 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" start-of-body= Oct 30 18:16:17.772387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.772290 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" podUID="7403bbb1-5c6e-431f-88a2-1005b4a22421" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.97:15021/healthz/ready\": dial tcp 10.133.0.97:15021: connect: connection refused" Oct 30 18:16:17.813081 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.813049 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92" Oct 30 18:16:17.822683 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.822653 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb"] Oct 30 18:16:17.837837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.837790 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/734d033e-b406-4d56-a09f-ac34c9bf0951-kube-api-access-clzt7\" (UniqueName: \"kubernetes.io/projected/734d033e-b406-4d56-a09f-ac34c9bf0951-kube-api-access-clzt7\") pod \"734d033e-b406-4d56-a09f-ac34c9bf0951\" (UID: \"734d033e-b406-4d56-a09f-ac34c9bf0951\") " Oct 30 18:16:17.838126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.838105 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/734d033e-b406-4d56-a09f-ac34c9bf0951-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734d033e-b406-4d56-a09f-ac34c9bf0951-maas-api-xks-ca\") pod \"734d033e-b406-4d56-a09f-ac34c9bf0951\" (UID: \"734d033e-b406-4d56-a09f-ac34c9bf0951\") " Oct 30 18:16:17.838283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.838265 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/734d033e-b406-4d56-a09f-ac34c9bf0951-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734d033e-b406-4d56-a09f-ac34c9bf0951-maas-api-openshift-ca\") pod \"734d033e-b406-4d56-a09f-ac34c9bf0951\" (UID: \"734d033e-b406-4d56-a09f-ac34c9bf0951\") " Oct 30 18:16:17.887581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.840471 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/734d033e-b406-4d56-a09f-ac34c9bf0951-maas-api-openshift-ca" pod "734d033e-b406-4d56-a09f-ac34c9bf0951" (UID: "734d033e-b406-4d56-a09f-ac34c9bf0951"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:17.887581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.841659 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" start-of-body= Oct 30 18:16:17.887581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.887345 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/734d033e-b406-4d56-a09f-ac34c9bf0951-maas-api-xks-ca" pod "734d033e-b406-4d56-a09f-ac34c9bf0951" (UID: "734d033e-b406-4d56-a09f-ac34c9bf0951"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:17.887581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.886025 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/734d033e-b406-4d56-a09f-ac34c9bf0951-kube-api-access-clzt7" pod "734d033e-b406-4d56-a09f-ac34c9bf0951" (UID: "734d033e-b406-4d56-a09f-ac34c9bf0951"). InnerVolumeSpecName "kube-api-access-clzt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:17.887581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.887387 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" podUID="f2ebd246-b899-4a1d-9067-3993742fb028" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.98:15021/healthz/ready\": dial tcp 10.133.0.98:15021: connect: connection refused" Oct 30 18:16:17.889308 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.889280 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-clzt7\" (UniqueName: \"kubernetes.io/projected/734d033e-b406-4d56-a09f-ac34c9bf0951-kube-api-access-clzt7\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:17.889406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.889312 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/734d033e-b406-4d56-a09f-ac34c9bf0951-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:17.889406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:17.889328 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/734d033e-b406-4d56-a09f-ac34c9bf0951-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:18.080676 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:18.080647 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd"] Oct 30 18:16:18.084709 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:16:18.084673 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd74ee1cf_7d30_49b6_880d_a7368794823d.slice/crio-b5555675f49deac9cb422e983d9792ce541e0d82c5a46f787f7651ff0d1c6f7a WatchSource:0}: Error finding container b5555675f49deac9cb422e983d9792ce541e0d82c5a46f787f7651ff0d1c6f7a: Status 404 returned error can't find the container with id b5555675f49deac9cb422e983d9792ce541e0d82c5a46f787f7651ff0d1c6f7a Oct 30 18:16:18.479442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:18.479401 5232 generic.go:356] "Generic (PLEG): container finished" podID="e8606381-4cb4-4096-9365-6e30d42a54ba" containerID="6f1ab5924188988bfc26f587192f5cfd604155e5ae8e79c9d9f81aff76107724" exitCode=0 Oct 30 18:16:18.479683 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:18.479649 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr" event={"ID":"e8606381-4cb4-4096-9365-6e30d42a54ba","Type":"ContainerDied","Data":"6f1ab5924188988bfc26f587192f5cfd604155e5ae8e79c9d9f81aff76107724"} Oct 30 18:16:18.482161 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:18.482129 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb" event={"ID":"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083","Type":"ContainerStarted","Data":"602f6536b61e823920f912d64ae57f8afd13ba3cfb3b8063c827cd9d1f4b6965"} Oct 30 18:16:18.482290 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:18.482184 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb" event={"ID":"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083","Type":"ContainerStarted","Data":"0d5b81b2fe8eb80d29740a0573e70de38511a9e8c7d2c07769e129f9e907297b"} Oct 30 18:16:18.484761 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:18.484732 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92" event={"ID":"734d033e-b406-4d56-a09f-ac34c9bf0951","Type":"ContainerDied","Data":"e40e8e2e353bf08b4530603b470b655d5ba147c821e9e59668987ca0b7097543"} Oct 30 18:16:18.484879 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:18.484779 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e40e8e2e353bf08b4530603b470b655d5ba147c821e9e59668987ca0b7097543" Oct 30 18:16:18.484879 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:18.484831 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92" Oct 30 18:16:18.500304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:18.496982 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" event={"ID":"d74ee1cf-7d30-49b6-880d-a7368794823d","Type":"ContainerStarted","Data":"56872a09f78362d8951e8a644657fdd967c7a63eb664b4d1715501850a3b6ff6"} Oct 30 18:16:18.500837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:18.500810 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" event={"ID":"d74ee1cf-7d30-49b6-880d-a7368794823d","Type":"ContainerStarted","Data":"b5555675f49deac9cb422e983d9792ce541e0d82c5a46f787f7651ff0d1c6f7a"} Oct 30 18:16:18.763615 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:18.725144 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:18.845705 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:18.815254 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb" podStartSLOduration=2.815231653 podStartE2EDuration="2.815231653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:18.578966412 +0000 UTC m=+877.895386732" watchObservedRunningTime="2026-10-30 18:16:18.815231653 +0000 UTC m=+878.131651974" Oct 30 18:16:18.845705 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:18.841301 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podStartSLOduration=2.84128151 podStartE2EDuration="2.84128151s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:18.697247899 +0000 UTC m=+878.013668221" watchObservedRunningTime="2026-10-30 18:16:18.84128151 +0000 UTC m=+878.157701831" Oct 30 18:16:19.222720 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.222685 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:19.224434 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.224367 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:19.225640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.225590 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:16:19.497208 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.497180 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:19.498961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.498926 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:19.500338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.500316 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:16:19.510294 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.510250 5232 generic.go:356] "Generic (PLEG): container finished" podID="9bd1fe1d-3eaf-4d4e-857e-caeb89e49083" containerID="602f6536b61e823920f912d64ae57f8afd13ba3cfb3b8063c827cd9d1f4b6965" exitCode=0 Oct 30 18:16:19.510716 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.510693 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb" event={"ID":"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083","Type":"ContainerDied","Data":"602f6536b61e823920f912d64ae57f8afd13ba3cfb3b8063c827cd9d1f4b6965"} Oct 30 18:16:19.539203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.539173 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:19.540882 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.539750 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:19.540882 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.540878 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:16:19.773410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.773330 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr" Oct 30 18:16:19.878231 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.878199 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e8606381-4cb4-4096-9365-6e30d42a54ba-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8606381-4cb4-4096-9365-6e30d42a54ba-maas-api-xks-ca\") pod \"e8606381-4cb4-4096-9365-6e30d42a54ba\" (UID: \"e8606381-4cb4-4096-9365-6e30d42a54ba\") " Oct 30 18:16:19.878713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.878332 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e8606381-4cb4-4096-9365-6e30d42a54ba-kube-api-access-27mt7\" (UniqueName: \"kubernetes.io/projected/e8606381-4cb4-4096-9365-6e30d42a54ba-kube-api-access-27mt7\") pod \"e8606381-4cb4-4096-9365-6e30d42a54ba\" (UID: \"e8606381-4cb4-4096-9365-6e30d42a54ba\") " Oct 30 18:16:19.878713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.878431 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e8606381-4cb4-4096-9365-6e30d42a54ba-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8606381-4cb4-4096-9365-6e30d42a54ba-maas-api-openshift-ca\") pod \"e8606381-4cb4-4096-9365-6e30d42a54ba\" (UID: \"e8606381-4cb4-4096-9365-6e30d42a54ba\") " Oct 30 18:16:19.878870 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.878839 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e8606381-4cb4-4096-9365-6e30d42a54ba-maas-api-openshift-ca" pod "e8606381-4cb4-4096-9365-6e30d42a54ba" (UID: "e8606381-4cb4-4096-9365-6e30d42a54ba"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:19.879154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.879128 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e8606381-4cb4-4096-9365-6e30d42a54ba-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:19.880750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.880723 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e8606381-4cb4-4096-9365-6e30d42a54ba-maas-api-xks-ca" pod "e8606381-4cb4-4096-9365-6e30d42a54ba" (UID: "e8606381-4cb4-4096-9365-6e30d42a54ba"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:19.880854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.880773 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e8606381-4cb4-4096-9365-6e30d42a54ba-kube-api-access-27mt7" pod "e8606381-4cb4-4096-9365-6e30d42a54ba" (UID: "e8606381-4cb4-4096-9365-6e30d42a54ba"). InnerVolumeSpecName "kube-api-access-27mt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:19.980103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.980068 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27mt7\" (UniqueName: \"kubernetes.io/projected/e8606381-4cb4-4096-9365-6e30d42a54ba-kube-api-access-27mt7\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:19.980103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:19.980098 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e8606381-4cb4-4096-9365-6e30d42a54ba-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:20.595969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:20.595937 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr" Oct 30 18:16:20.598104 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:20.597490 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr" event={"ID":"e8606381-4cb4-4096-9365-6e30d42a54ba","Type":"ContainerDied","Data":"82e4baf2f287cc165d81ad01bc0d7606e71fb20ace7273b519a4a9f8d1f7306f"} Oct 30 18:16:20.598104 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:20.597585 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="82e4baf2f287cc165d81ad01bc0d7606e71fb20ace7273b519a4a9f8d1f7306f" Oct 30 18:16:20.800979 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:20.800945 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb" Oct 30 18:16:20.897602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:20.897472 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-kube-api-access-ftskj\" (UniqueName: \"kubernetes.io/projected/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-kube-api-access-ftskj\") pod \"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083\" (UID: \"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083\") " Oct 30 18:16:20.898016 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:20.897815 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-maas-api-xks-ca\") pod \"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083\" (UID: \"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083\") " Oct 30 18:16:20.898016 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:20.897931 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-maas-api-openshift-ca\") pod \"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083\" (UID: \"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083\") " Oct 30 18:16:20.898568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:20.898537 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-maas-api-openshift-ca" pod "9bd1fe1d-3eaf-4d4e-857e-caeb89e49083" (UID: "9bd1fe1d-3eaf-4d4e-857e-caeb89e49083"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:20.898793 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:20.898772 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:20.899897 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:20.899877 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-kube-api-access-ftskj" pod "9bd1fe1d-3eaf-4d4e-857e-caeb89e49083" (UID: "9bd1fe1d-3eaf-4d4e-857e-caeb89e49083"). InnerVolumeSpecName "kube-api-access-ftskj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:20.900113 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:20.900093 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-maas-api-xks-ca" pod "9bd1fe1d-3eaf-4d4e-857e-caeb89e49083" (UID: "9bd1fe1d-3eaf-4d4e-857e-caeb89e49083"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:21.000579 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:21.000495 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:21.000579 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:21.000579 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftskj\" (UniqueName: \"kubernetes.io/projected/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083-kube-api-access-ftskj\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:21.605570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:21.605539 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb" Oct 30 18:16:21.605768 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:21.605538 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb" event={"ID":"9bd1fe1d-3eaf-4d4e-857e-caeb89e49083","Type":"ContainerDied","Data":"0d5b81b2fe8eb80d29740a0573e70de38511a9e8c7d2c07769e129f9e907297b"} Oct 30 18:16:21.605768 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:21.605673 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d5b81b2fe8eb80d29740a0573e70de38511a9e8c7d2c07769e129f9e907297b" Oct 30 18:16:27.043650 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:16:27.043609 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:16:27.721693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:27.721656 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:27.734735 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:27.734712 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:28.670471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:28.670440 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:36.293529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:36.289105 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:36.382664 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:36.382612 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:36.490700 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:36.489617 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:36.673735 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:36.673624 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:36.884706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:36.884663 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.175545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:37.172017 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.225539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:37.221617 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.282698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:37.282645 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.291653 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:37.291604 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.292291 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:37.291913 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.293487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:37.292890 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:16:37.312530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:37.308459 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.317949 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:37.313112 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.405537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:37.400611 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.405537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:37.401973 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:16:37.500595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:37.474990 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Oct 30 18:16:37.657545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:37.579628 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded" Oct 30 18:16:37.657545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:37.609831 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:16:37.657545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:37.639949 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:37.747709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:37.699386 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:37.748489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:37.748461 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:37.924989 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:37.923844 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:16:37.975544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:37.964646 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:16:38.196116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.196007 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:16:38.198438 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.198409 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:16:38.216581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.199664 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:38.240660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.223926 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:16:38.244592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.244555 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:16:38.304751 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.304700 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:38.310698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.310434 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:16:38.310982 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.310551 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:38.327607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.321733 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:38.328663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.328636 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:16:38.369605 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.344269 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:38.370087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.370065 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:16:38.373160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.373048 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:16:38.373381 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.373354 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:16:38.424852 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.424807 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:38.698538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.640597 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:38.761641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.732609 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:38.762336 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.762299 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:38.969597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:38.940058 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.233729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:39.233620 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.251542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:39.247810 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.355643 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:39.328655 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.437534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:39.328697 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.437534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:39.328734 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.437534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:39.328754 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.437534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:39.349960 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.437534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:39.381663 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.437534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:39.397307 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.468716 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:39.468611 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 18:16:39.578535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:39.577626 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:39.578535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:39.577700 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" podUID="3480923d-8625-430e-819c-628e203c4946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.611542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:39.610645 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:39.611542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:39.610725 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" podUID="f8c0e348-2b8d-48e2-9ec0-a21a83db391e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.776877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:39.776825 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:16:39.915183 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:39.859116 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:16:39.950849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:39.950796 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:39.983864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:39.983817 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:40.095543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:40.059413 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:40.095543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:40.090075 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": context deadline exceeded" Oct 30 18:16:40.122544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:40.120042 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:16:40.122544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:40.120196 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" podUID="ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.32:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:40.314538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:40.259831 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:40.480688 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:40.480638 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:40.508470 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:40.491046 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:40.767599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:40.747706 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:40.767599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:40.748557 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:16:40.767599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:40.749816 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:16:40.874536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:40.850021 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:16:41.042594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:40.876909 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:16:41.042594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:40.958841 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:41.081691 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:41.081637 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:16:41.358106 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:41.145673 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:16:41.358620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:41.259100 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Oct 30 18:16:41.358813 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:41.331635 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:41.517990 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:41.517938 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:41.763204 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:41.763154 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:16:41.763576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:41.763549 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:16:41.766406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:41.763102 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:42.074585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.057724 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:16:42.373288 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.373207 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:16:42.373587 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.373491 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247"] Oct 30 18:16:42.374632 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.374609 5232 cpu_manager.go:414] "containerMap: removing container" podUID="9bd1fe1d-3eaf-4d4e-857e-caeb89e49083" containerName="revoke-keys" Oct 30 18:16:42.385886 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.385844 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9bd1fe1d-3eaf-4d4e-857e-caeb89e49083" containerName="revoke-keys" Oct 30 18:16:42.386026 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.385931 5232 cpu_manager.go:414] "containerMap: removing container" podUID="e8606381-4cb4-4096-9365-6e30d42a54ba" containerName="revoke-keys" Oct 30 18:16:42.386026 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.385944 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e8606381-4cb4-4096-9365-6e30d42a54ba" containerName="revoke-keys" Oct 30 18:16:42.386026 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.385962 5232 cpu_manager.go:414] "containerMap: removing container" podUID="734d033e-b406-4d56-a09f-ac34c9bf0951" containerName="revoke-keys" Oct 30 18:16:42.386026 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.385970 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="734d033e-b406-4d56-a09f-ac34c9bf0951" containerName="revoke-keys" Oct 30 18:16:42.386243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.386209 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="9bd1fe1d-3eaf-4d4e-857e-caeb89e49083" containerName="revoke-keys" Oct 30 18:16:42.386293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.386257 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="e8606381-4cb4-4096-9365-6e30d42a54ba" containerName="revoke-keys" Oct 30 18:16:42.386293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.386268 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="734d033e-b406-4d56-a09f-ac34c9bf0951" containerName="revoke-keys" Oct 30 18:16:42.473281 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.473239 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5"] Oct 30 18:16:42.474656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.474629 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.487986 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.487946 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247"] Oct 30 18:16:42.488151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.487995 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5"] Oct 30 18:16:42.488151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.488062 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:16:42.488151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.488113 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.488330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.488177 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:16:42.488330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.488244 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:16:42.488330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.488279 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:16:42.488330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.488311 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:16:42.488568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.488341 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:16:42.488568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.488374 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:16:42.488568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.488405 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:16:42.488568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.488437 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:16:42.662955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.662872 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.662955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.662909 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.662955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.662934 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.663486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.663027 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f08f06-6d9f-4003-b35a-62de857fbe80-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.663486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.663129 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.663486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.663165 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.663755 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.663702 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.663842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.663779 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.663842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.663798 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qfbm\" (UniqueName: \"kubernetes.io/projected/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-kube-api-access-9qfbm\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.663956 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.663907 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.664016 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.663965 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.664072 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.664014 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.664126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.664098 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.664177 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.664151 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.664230 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.664211 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.664272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.664255 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.664376 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.664360 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cj92j\" (UniqueName: \"kubernetes.io/projected/31f08f06-6d9f-4003-b35a-62de857fbe80-kube-api-access-cj92j\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.664483 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.664467 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.766557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.764919 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm"] Oct 30 18:16:42.766557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.765959 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.766557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.766002 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.766557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.766043 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.766557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.766077 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.766557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.766116 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.766557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.766155 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.766557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.766214 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cj92j\" (UniqueName: \"kubernetes.io/projected/31f08f06-6d9f-4003-b35a-62de857fbe80-kube-api-access-cj92j\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.766557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.766278 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.766557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.766366 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.766557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.766393 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.766557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.766429 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.766557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.766466 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f08f06-6d9f-4003-b35a-62de857fbe80-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.766557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.766544 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.766557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.766570 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.767437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.766673 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.767437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.766749 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.767437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.766773 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qfbm\" (UniqueName: \"kubernetes.io/projected/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-kube-api-access-9qfbm\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.767437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.766840 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.767437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.767078 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.767437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.767192 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-credential-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.772940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.767444 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.772940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.768200 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.772940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.768597 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.772940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.766669 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-data\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.772940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.768880 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f08f06-6d9f-4003-b35a-62de857fbe80-istiod-ca-cert\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.772940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.769127 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.772940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.769349 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-workload-socket\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.772940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.769602 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-workload-certs\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.772940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.771288 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.772940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.771563 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm" Oct 30 18:16:42.772940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.772370 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.777180 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.777154 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-envoy\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.780227 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.780178 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-podinfo\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.786607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.786051 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:16:42.786607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.786118 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:16:42.786826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.786778 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm"] Oct 30 18:16:42.788419 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.788391 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.789617 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.789593 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cj92j\" (UniqueName: \"kubernetes.io/projected/31f08f06-6d9f-4003-b35a-62de857fbe80-kube-api-access-cj92j\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.790072 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.790053 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-token\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.792881 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.792842 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qfbm\" (UniqueName: \"kubernetes.io/projected/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-kube-api-access-9qfbm\") pod \"e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.807237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.807214 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:42.821583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.819357 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:42.874087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.870184 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ff96556-c6wlm\" (UID: \"7ff9c66e-0e88-4ce4-9907-09cbd81264ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm" Oct 30 18:16:42.874087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.870621 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ff96556-c6wlm\" (UID: \"7ff9c66e-0e88-4ce4-9907-09cbd81264ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm" Oct 30 18:16:42.874087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.870765 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mjktz\" (UniqueName: \"kubernetes.io/projected/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-kube-api-access-mjktz\") pod \"maas-api-revoke-sub-4ff96556-c6wlm\" (UID: \"7ff9c66e-0e88-4ce4-9907-09cbd81264ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm" Oct 30 18:16:42.874087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.871951 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z"] Oct 30 18:16:42.897601 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.890075 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z" Oct 30 18:16:42.900773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.900492 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z"] Oct 30 18:16:42.972268 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.972229 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75059e0c-g794z\" (UID: \"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z" Oct 30 18:16:42.972437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.972297 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75059e0c-g794z\" (UID: \"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z" Oct 30 18:16:42.972437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.972428 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ff96556-c6wlm\" (UID: \"7ff9c66e-0e88-4ce4-9907-09cbd81264ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm" Oct 30 18:16:42.972618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.972604 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ff96556-c6wlm\" (UID: \"7ff9c66e-0e88-4ce4-9907-09cbd81264ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm" Oct 30 18:16:42.972719 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.972700 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mjktz\" (UniqueName: \"kubernetes.io/projected/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-kube-api-access-mjktz\") pod \"maas-api-revoke-sub-4ff96556-c6wlm\" (UID: \"7ff9c66e-0e88-4ce4-9907-09cbd81264ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm" Oct 30 18:16:42.972798 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.972784 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-282f2\" (UniqueName: \"kubernetes.io/projected/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-kube-api-access-282f2\") pod \"maas-api-revoke-sub-75059e0c-g794z\" (UID: \"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z" Oct 30 18:16:42.974752 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.974725 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ff96556-c6wlm\" (UID: \"7ff9c66e-0e88-4ce4-9907-09cbd81264ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm" Oct 30 18:16:42.976935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.976904 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ff96556-c6wlm\" (UID: \"7ff9c66e-0e88-4ce4-9907-09cbd81264ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm" Oct 30 18:16:42.993575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:42.993540 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mjktz\" (UniqueName: \"kubernetes.io/projected/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-kube-api-access-mjktz\") pod \"maas-api-revoke-sub-4ff96556-c6wlm\" (UID: \"7ff9c66e-0e88-4ce4-9907-09cbd81264ea\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm" Oct 30 18:16:43.096731 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.094567 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75059e0c-g794z\" (UID: \"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z" Oct 30 18:16:43.096731 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.094924 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-282f2\" (UniqueName: \"kubernetes.io/projected/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-kube-api-access-282f2\") pod \"maas-api-revoke-sub-75059e0c-g794z\" (UID: \"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z" Oct 30 18:16:43.096731 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.094988 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75059e0c-g794z\" (UID: \"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z" Oct 30 18:16:43.097373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.097299 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75059e0c-g794z\" (UID: \"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z" Oct 30 18:16:43.163640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.162171 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75059e0c-g794z\" (UID: \"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z" Oct 30 18:16:43.210901 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.210805 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-282f2\" (UniqueName: \"kubernetes.io/projected/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-kube-api-access-282f2\") pod \"maas-api-revoke-sub-75059e0c-g794z\" (UID: \"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z" Oct 30 18:16:43.210901 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.210859 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm" Oct 30 18:16:43.220484 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.219287 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z" Oct 30 18:16:43.254496 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:16:43.254460 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf0634c2b_f6d4_4d94_88d3_8afb1b76b3d0.slice/crio-46ea0c3b3670f4824c0d294e6dcc42d1422b86f6a1718bbc4c09050997c78ef1 WatchSource:0}: Error finding container 46ea0c3b3670f4824c0d294e6dcc42d1422b86f6a1718bbc4c09050997c78ef1: Status 404 returned error can't find the container with id 46ea0c3b3670f4824c0d294e6dcc42d1422b86f6a1718bbc4c09050997c78ef1 Oct 30 18:16:43.256540 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:16:43.256479 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31f08f06_6d9f_4003_b35a_62de857fbe80.slice/crio-9c6a997df3b8e911737fe1ed8c4cd77ddd16576e8a2d044f49d52ebb3dab9d89 WatchSource:0}: Error finding container 9c6a997df3b8e911737fe1ed8c4cd77ddd16576e8a2d044f49d52ebb3dab9d89: Status 404 returned error can't find the container with id 9c6a997df3b8e911737fe1ed8c4cd77ddd16576e8a2d044f49d52ebb3dab9d89 Oct 30 18:16:43.258988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.258265 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:16:43.258988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.258349 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:16:43.258988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.258394 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:16:43.258988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.258804 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247"] Oct 30 18:16:43.259230 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.259100 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:16:43.259230 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.259172 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:16:43.259230 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.259215 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:16:43.280077 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.274725 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5"] Oct 30 18:16:43.544913 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:16:43.544872 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ff9c66e_0e88_4ce4_9907_09cbd81264ea.slice/crio-d6243e0fd612807228364c4fd880f72d9987666d1dbddaf60bc90407cb0a8f32 WatchSource:0}: Error finding container d6243e0fd612807228364c4fd880f72d9987666d1dbddaf60bc90407cb0a8f32: Status 404 returned error can't find the container with id d6243e0fd612807228364c4fd880f72d9987666d1dbddaf60bc90407cb0a8f32 Oct 30 18:16:43.546422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.546390 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm"] Oct 30 18:16:43.557732 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.557228 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z"] Oct 30 18:16:43.977095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.977003 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z" event={"ID":"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8","Type":"ContainerStarted","Data":"cff94f5d33b62fc97d1ef856bebdc4dd54575acdceddd0d1ecca4ed6fc440cbb"} Oct 30 18:16:43.977095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.977062 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z" event={"ID":"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8","Type":"ContainerStarted","Data":"aaad02bff60124f21f114dba9de17256cf0e68bc378686452ef22a72dc9d34de"} Oct 30 18:16:43.978708 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.978684 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm" event={"ID":"7ff9c66e-0e88-4ce4-9907-09cbd81264ea","Type":"ContainerStarted","Data":"d6243e0fd612807228364c4fd880f72d9987666d1dbddaf60bc90407cb0a8f32"} Oct 30 18:16:43.980855 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.980567 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" event={"ID":"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0","Type":"ContainerStarted","Data":"7e52361b9c056affff55f26709a0ea054c70cd21d17787aa1e947e64eb17c278"} Oct 30 18:16:43.980855 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.980613 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" event={"ID":"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0","Type":"ContainerStarted","Data":"46ea0c3b3670f4824c0d294e6dcc42d1422b86f6a1718bbc4c09050997c78ef1"} Oct 30 18:16:43.982164 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.982136 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" event={"ID":"31f08f06-6d9f-4003-b35a-62de857fbe80","Type":"ContainerStarted","Data":"ce86ab799b5b32e654b04d184f606cafb1c99b890e0d9558fe170b221ea81793"} Oct 30 18:16:43.982270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:43.982181 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" event={"ID":"31f08f06-6d9f-4003-b35a-62de857fbe80","Type":"ContainerStarted","Data":"9c6a997df3b8e911737fe1ed8c4cd77ddd16576e8a2d044f49d52ebb3dab9d89"} Oct 30 18:16:44.093181 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:44.093079 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z" podStartSLOduration=2.093065322 podStartE2EDuration="2.093065322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:44.038611408 +0000 UTC m=+903.355031730" watchObservedRunningTime="2026-10-30 18:16:44.093065322 +0000 UTC m=+903.409485645" Oct 30 18:16:44.146534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:44.133422 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" podStartSLOduration=5.133400268 podStartE2EDuration="5.133400268s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:44.131024622 +0000 UTC m=+903.447444943" watchObservedRunningTime="2026-10-30 18:16:44.133400268 +0000 UTC m=+903.449820590" Oct 30 18:16:44.149671 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:44.148422 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" podStartSLOduration=5.148402939 podStartE2EDuration="5.148402939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:44.090577617 +0000 UTC m=+903.406997939" watchObservedRunningTime="2026-10-30 18:16:44.148402939 +0000 UTC m=+903.464823262" Oct 30 18:16:44.858672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:44.858631 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 18:16:44.858940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:44.858914 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" podUID="f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 18:16:44.859240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:44.858631 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:16:44.859375 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:44.859354 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" podUID="31f08f06-6d9f-4003-b35a-62de857fbe80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 18:16:44.918615 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:44.915799 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:44.918615 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:44.915863 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:44.994954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:44.994900 5232 generic.go:356] "Generic (PLEG): container finished" podID="c30bde54-a1ab-4ab9-86e1-b3248a4e73b8" containerID="cff94f5d33b62fc97d1ef856bebdc4dd54575acdceddd0d1ecca4ed6fc440cbb" exitCode=0 Oct 30 18:16:44.995416 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:44.995060 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z" event={"ID":"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8","Type":"ContainerDied","Data":"cff94f5d33b62fc97d1ef856bebdc4dd54575acdceddd0d1ecca4ed6fc440cbb"} Oct 30 18:16:44.997314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:44.997289 5232 generic.go:356] "Generic (PLEG): container finished" podID="7ff9c66e-0e88-4ce4-9907-09cbd81264ea" containerID="eb1436ad530ca299796aa72cb7eed54770fcb2a48a72ab4a3c8fbd62788e4a5f" exitCode=0 Oct 30 18:16:44.997460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:44.997370 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm" event={"ID":"7ff9c66e-0e88-4ce4-9907-09cbd81264ea","Type":"ContainerDied","Data":"eb1436ad530ca299796aa72cb7eed54770fcb2a48a72ab4a3c8fbd62788e4a5f"} Oct 30 18:16:45.582780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:45.582745 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2"] Oct 30 18:16:45.589467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:45.589437 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2" Oct 30 18:16:45.591283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:45.591262 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2"] Oct 30 18:16:45.664762 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:45.664723 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4f754f7-41c2-48f0-9236-b27c52d968f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db839263-8g9c2\" (UID: \"f4f754f7-41c2-48f0-9236-b27c52d968f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2" Oct 30 18:16:45.665372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:45.665346 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4f754f7-41c2-48f0-9236-b27c52d968f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db839263-8g9c2\" (UID: \"f4f754f7-41c2-48f0-9236-b27c52d968f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2" Oct 30 18:16:45.665679 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:45.665644 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t579x\" (UniqueName: \"kubernetes.io/projected/f4f754f7-41c2-48f0-9236-b27c52d968f2-kube-api-access-t579x\") pod \"maas-api-revoke-sub-db839263-8g9c2\" (UID: \"f4f754f7-41c2-48f0-9236-b27c52d968f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2" Oct 30 18:16:45.767652 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:45.767617 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4f754f7-41c2-48f0-9236-b27c52d968f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db839263-8g9c2\" (UID: \"f4f754f7-41c2-48f0-9236-b27c52d968f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2" Oct 30 18:16:45.767832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:45.767697 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t579x\" (UniqueName: \"kubernetes.io/projected/f4f754f7-41c2-48f0-9236-b27c52d968f2-kube-api-access-t579x\") pod \"maas-api-revoke-sub-db839263-8g9c2\" (UID: \"f4f754f7-41c2-48f0-9236-b27c52d968f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2" Oct 30 18:16:45.767832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:45.767792 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4f754f7-41c2-48f0-9236-b27c52d968f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db839263-8g9c2\" (UID: \"f4f754f7-41c2-48f0-9236-b27c52d968f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2" Oct 30 18:16:45.768881 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:45.768853 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4f754f7-41c2-48f0-9236-b27c52d968f2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-db839263-8g9c2\" (UID: \"f4f754f7-41c2-48f0-9236-b27c52d968f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2" Oct 30 18:16:45.770918 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:45.770889 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4f754f7-41c2-48f0-9236-b27c52d968f2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-db839263-8g9c2\" (UID: \"f4f754f7-41c2-48f0-9236-b27c52d968f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2" Oct 30 18:16:45.786948 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:45.786921 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t579x\" (UniqueName: \"kubernetes.io/projected/f4f754f7-41c2-48f0-9236-b27c52d968f2-kube-api-access-t579x\") pod \"maas-api-revoke-sub-db839263-8g9c2\" (UID: \"f4f754f7-41c2-48f0-9236-b27c52d968f2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2" Oct 30 18:16:45.821568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:45.809474 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:16:45.826652 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:45.826611 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" podUID="31f08f06-6d9f-4003-b35a-62de857fbe80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 18:16:45.833680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:45.824902 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 18:16:45.838562 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:45.833849 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" podUID="f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 18:16:45.905551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:45.905498 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2" Oct 30 18:16:46.294415 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.292838 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2"] Oct 30 18:16:46.595694 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.595666 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z" Oct 30 18:16:46.597488 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.597466 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm" Oct 30 18:16:46.742536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.713934 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-maas-api-openshift-ca\") pod \"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8\" (UID: \"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8\") " Oct 30 18:16:46.742536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.714047 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-kube-api-access-282f2\" (UniqueName: \"kubernetes.io/projected/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-kube-api-access-282f2\") pod \"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8\" (UID: \"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8\") " Oct 30 18:16:46.742536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.714191 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-maas-api-openshift-ca\") pod \"7ff9c66e-0e88-4ce4-9907-09cbd81264ea\" (UID: \"7ff9c66e-0e88-4ce4-9907-09cbd81264ea\") " Oct 30 18:16:46.742536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.714263 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-maas-api-xks-ca\") pod \"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8\" (UID: \"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8\") " Oct 30 18:16:46.742536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.714488 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-maas-api-xks-ca\") pod \"7ff9c66e-0e88-4ce4-9907-09cbd81264ea\" (UID: \"7ff9c66e-0e88-4ce4-9907-09cbd81264ea\") " Oct 30 18:16:46.742536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.714561 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-kube-api-access-mjktz\" (UniqueName: \"kubernetes.io/projected/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-kube-api-access-mjktz\") pod \"7ff9c66e-0e88-4ce4-9907-09cbd81264ea\" (UID: \"7ff9c66e-0e88-4ce4-9907-09cbd81264ea\") " Oct 30 18:16:46.826891 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.783853 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-maas-api-openshift-ca" pod "7ff9c66e-0e88-4ce4-9907-09cbd81264ea" (UID: "7ff9c66e-0e88-4ce4-9907-09cbd81264ea"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:46.826891 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.784229 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-maas-api-openshift-ca" pod "c30bde54-a1ab-4ab9-86e1-b3248a4e73b8" (UID: "c30bde54-a1ab-4ab9-86e1-b3248a4e73b8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:46.849565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.849156 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:46.849565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.849191 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:46.852551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.852039 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:16:46.852551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.852092 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" podUID="31f08f06-6d9f-4003-b35a-62de857fbe80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 18:16:46.852551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.852491 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 18:16:46.852827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.852554 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" podUID="f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 18:16:46.907532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.907087 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-kube-api-access-282f2" pod "c30bde54-a1ab-4ab9-86e1-b3248a4e73b8" (UID: "c30bde54-a1ab-4ab9-86e1-b3248a4e73b8"). InnerVolumeSpecName "kube-api-access-282f2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:46.907732 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.907698 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-maas-api-xks-ca" pod "c30bde54-a1ab-4ab9-86e1-b3248a4e73b8" (UID: "c30bde54-a1ab-4ab9-86e1-b3248a4e73b8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:46.909144 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.909115 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-kube-api-access-mjktz" pod "7ff9c66e-0e88-4ce4-9907-09cbd81264ea" (UID: "7ff9c66e-0e88-4ce4-9907-09cbd81264ea"). InnerVolumeSpecName "kube-api-access-mjktz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:46.909877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.909546 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-maas-api-xks-ca" pod "7ff9c66e-0e88-4ce4-9907-09cbd81264ea" (UID: "7ff9c66e-0e88-4ce4-9907-09cbd81264ea"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:46.957452 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.957415 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:46.957452 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.957452 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:46.957750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.957468 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mjktz\" (UniqueName: \"kubernetes.io/projected/7ff9c66e-0e88-4ce4-9907-09cbd81264ea-kube-api-access-mjktz\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:46.957750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:46.957484 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-282f2\" (UniqueName: \"kubernetes.io/projected/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8-kube-api-access-282f2\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:47.066025 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:47.061002 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2" event={"ID":"f4f754f7-41c2-48f0-9236-b27c52d968f2","Type":"ContainerStarted","Data":"5e13dfe252a3e1ebbfb26e11681e7b8247616b0ee0fe1c03b5b687628ea40e5e"} Oct 30 18:16:47.066025 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:47.061075 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2" event={"ID":"f4f754f7-41c2-48f0-9236-b27c52d968f2","Type":"ContainerStarted","Data":"707ff10c0ab2f34e7a64517cb07aff1a420752ebace29a4abacd1ee5704555bf"} Oct 30 18:16:47.066025 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:47.064302 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z" event={"ID":"c30bde54-a1ab-4ab9-86e1-b3248a4e73b8","Type":"ContainerDied","Data":"aaad02bff60124f21f114dba9de17256cf0e68bc378686452ef22a72dc9d34de"} Oct 30 18:16:47.066025 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:47.064352 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aaad02bff60124f21f114dba9de17256cf0e68bc378686452ef22a72dc9d34de" Oct 30 18:16:47.066025 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:47.064411 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z" Oct 30 18:16:47.067032 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:47.066990 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm" event={"ID":"7ff9c66e-0e88-4ce4-9907-09cbd81264ea","Type":"ContainerDied","Data":"d6243e0fd612807228364c4fd880f72d9987666d1dbddaf60bc90407cb0a8f32"} Oct 30 18:16:47.067148 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:47.067051 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6243e0fd612807228364c4fd880f72d9987666d1dbddaf60bc90407cb0a8f32" Oct 30 18:16:47.067148 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:47.067128 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm" Oct 30 18:16:47.102235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:47.102136 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2" podStartSLOduration=2.102120172 podStartE2EDuration="2.102120172s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:47.099584716 +0000 UTC m=+906.416005038" watchObservedRunningTime="2026-10-30 18:16:47.102120172 +0000 UTC m=+906.418540494" Oct 30 18:16:47.809345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:47.809309 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:16:47.809816 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:47.809371 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" podUID="31f08f06-6d9f-4003-b35a-62de857fbe80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 18:16:47.829545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:47.824071 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 18:16:47.829831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:47.829794 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" podUID="f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 18:16:48.081760 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:48.081725 5232 generic.go:356] "Generic (PLEG): container finished" podID="f4f754f7-41c2-48f0-9236-b27c52d968f2" containerID="5e13dfe252a3e1ebbfb26e11681e7b8247616b0ee0fe1c03b5b687628ea40e5e" exitCode=0 Oct 30 18:16:48.082006 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:48.081879 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2" event={"ID":"f4f754f7-41c2-48f0-9236-b27c52d968f2","Type":"ContainerDied","Data":"5e13dfe252a3e1ebbfb26e11681e7b8247616b0ee0fe1c03b5b687628ea40e5e"} Oct 30 18:16:48.819961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:48.819924 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:16:48.820965 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:48.820934 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" podUID="31f08f06-6d9f-4003-b35a-62de857fbe80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 18:16:48.821496 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:48.820863 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 18:16:48.831809 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:48.831779 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" podUID="f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 18:16:49.403863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:49.403837 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2" Oct 30 18:16:49.506029 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:49.505995 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4f754f7-41c2-48f0-9236-b27c52d968f2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4f754f7-41c2-48f0-9236-b27c52d968f2-maas-api-xks-ca\") pod \"f4f754f7-41c2-48f0-9236-b27c52d968f2\" (UID: \"f4f754f7-41c2-48f0-9236-b27c52d968f2\") " Oct 30 18:16:49.506448 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:49.506426 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4f754f7-41c2-48f0-9236-b27c52d968f2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4f754f7-41c2-48f0-9236-b27c52d968f2-maas-api-openshift-ca\") pod \"f4f754f7-41c2-48f0-9236-b27c52d968f2\" (UID: \"f4f754f7-41c2-48f0-9236-b27c52d968f2\") " Oct 30 18:16:49.506551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:49.506534 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4f754f7-41c2-48f0-9236-b27c52d968f2-kube-api-access-t579x\" (UniqueName: \"kubernetes.io/projected/f4f754f7-41c2-48f0-9236-b27c52d968f2-kube-api-access-t579x\") pod \"f4f754f7-41c2-48f0-9236-b27c52d968f2\" (UID: \"f4f754f7-41c2-48f0-9236-b27c52d968f2\") " Oct 30 18:16:49.506920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:49.506896 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4f754f7-41c2-48f0-9236-b27c52d968f2-maas-api-openshift-ca" pod "f4f754f7-41c2-48f0-9236-b27c52d968f2" (UID: "f4f754f7-41c2-48f0-9236-b27c52d968f2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:49.507327 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:49.507303 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4f754f7-41c2-48f0-9236-b27c52d968f2-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:49.508668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:49.508646 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4f754f7-41c2-48f0-9236-b27c52d968f2-maas-api-xks-ca" pod "f4f754f7-41c2-48f0-9236-b27c52d968f2" (UID: "f4f754f7-41c2-48f0-9236-b27c52d968f2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:49.508815 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:49.508793 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4f754f7-41c2-48f0-9236-b27c52d968f2-kube-api-access-t579x" pod "f4f754f7-41c2-48f0-9236-b27c52d968f2" (UID: "f4f754f7-41c2-48f0-9236-b27c52d968f2"). InnerVolumeSpecName "kube-api-access-t579x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:49.608806 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:49.608774 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4f754f7-41c2-48f0-9236-b27c52d968f2-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:49.608806 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:49.608803 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t579x\" (UniqueName: \"kubernetes.io/projected/f4f754f7-41c2-48f0-9236-b27c52d968f2-kube-api-access-t579x\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:49.809142 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:49.809107 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" start-of-body= Oct 30 18:16:49.809337 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:49.809164 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" podUID="31f08f06-6d9f-4003-b35a-62de857fbe80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": dial tcp 10.133.0.104:15021: connect: connection refused" Oct 30 18:16:49.820491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:49.820464 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" start-of-body= Oct 30 18:16:49.820984 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:49.820526 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" podUID="f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.105:15021/healthz/ready\": dial tcp 10.133.0.105:15021: connect: connection refused" Oct 30 18:16:50.101868 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:50.101775 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2" event={"ID":"f4f754f7-41c2-48f0-9236-b27c52d968f2","Type":"ContainerDied","Data":"707ff10c0ab2f34e7a64517cb07aff1a420752ebace29a4abacd1ee5704555bf"} Oct 30 18:16:50.101868 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:50.101811 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2" Oct 30 18:16:50.101868 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:50.101831 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="707ff10c0ab2f34e7a64517cb07aff1a420752ebace29a4abacd1ee5704555bf" Oct 30 18:16:51.027417 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.027250 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:51.028664 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.028633 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:51.030006 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.029967 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:16:51.045848 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.045820 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:51.046413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.046389 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:51.047853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.047833 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:16:51.783488 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.783458 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb"] Oct 30 18:16:51.784124 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.784108 5232 cpu_manager.go:414] "containerMap: removing container" podUID="c30bde54-a1ab-4ab9-86e1-b3248a4e73b8" containerName="revoke-keys" Oct 30 18:16:51.784196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.784125 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c30bde54-a1ab-4ab9-86e1-b3248a4e73b8" containerName="revoke-keys" Oct 30 18:16:51.784196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.784147 5232 cpu_manager.go:414] "containerMap: removing container" podUID="f4f754f7-41c2-48f0-9236-b27c52d968f2" containerName="revoke-keys" Oct 30 18:16:51.784196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.784152 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4f754f7-41c2-48f0-9236-b27c52d968f2" containerName="revoke-keys" Oct 30 18:16:51.784196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.784163 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7ff9c66e-0e88-4ce4-9907-09cbd81264ea" containerName="revoke-keys" Oct 30 18:16:51.784196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.784168 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7ff9c66e-0e88-4ce4-9907-09cbd81264ea" containerName="revoke-keys" Oct 30 18:16:51.784444 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.784264 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="c30bde54-a1ab-4ab9-86e1-b3248a4e73b8" containerName="revoke-keys" Oct 30 18:16:51.784444 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.784274 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4f754f7-41c2-48f0-9236-b27c52d968f2" containerName="revoke-keys" Oct 30 18:16:51.784444 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.784282 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7ff9c66e-0e88-4ce4-9907-09cbd81264ea" containerName="revoke-keys" Oct 30 18:16:51.788415 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.788396 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb" Oct 30 18:16:51.805064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.803237 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:16:51.805064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.803354 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:16:51.806875 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.806853 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb"] Oct 30 18:16:51.850483 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.850444 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18f5382-cae5-42b9-9b10-b7e6555fd628-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b85a6c82-n64pb\" (UID: \"c18f5382-cae5-42b9-9b10-b7e6555fd628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb" Oct 30 18:16:51.850648 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.850544 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6tsj2\" (UniqueName: \"kubernetes.io/projected/c18f5382-cae5-42b9-9b10-b7e6555fd628-kube-api-access-6tsj2\") pod \"maas-api-revoke-sub-b85a6c82-n64pb\" (UID: \"c18f5382-cae5-42b9-9b10-b7e6555fd628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb" Oct 30 18:16:51.850648 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.850607 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18f5382-cae5-42b9-9b10-b7e6555fd628-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b85a6c82-n64pb\" (UID: \"c18f5382-cae5-42b9-9b10-b7e6555fd628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb" Oct 30 18:16:51.952373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.952339 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18f5382-cae5-42b9-9b10-b7e6555fd628-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b85a6c82-n64pb\" (UID: \"c18f5382-cae5-42b9-9b10-b7e6555fd628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb" Oct 30 18:16:51.952552 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.952388 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6tsj2\" (UniqueName: \"kubernetes.io/projected/c18f5382-cae5-42b9-9b10-b7e6555fd628-kube-api-access-6tsj2\") pod \"maas-api-revoke-sub-b85a6c82-n64pb\" (UID: \"c18f5382-cae5-42b9-9b10-b7e6555fd628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb" Oct 30 18:16:51.952552 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.952427 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18f5382-cae5-42b9-9b10-b7e6555fd628-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b85a6c82-n64pb\" (UID: \"c18f5382-cae5-42b9-9b10-b7e6555fd628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb" Oct 30 18:16:51.953090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.953062 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18f5382-cae5-42b9-9b10-b7e6555fd628-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b85a6c82-n64pb\" (UID: \"c18f5382-cae5-42b9-9b10-b7e6555fd628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb" Oct 30 18:16:51.954903 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.954885 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18f5382-cae5-42b9-9b10-b7e6555fd628-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b85a6c82-n64pb\" (UID: \"c18f5382-cae5-42b9-9b10-b7e6555fd628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb" Oct 30 18:16:51.971850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:51.971821 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6tsj2\" (UniqueName: \"kubernetes.io/projected/c18f5382-cae5-42b9-9b10-b7e6555fd628-kube-api-access-6tsj2\") pod \"maas-api-revoke-sub-b85a6c82-n64pb\" (UID: \"c18f5382-cae5-42b9-9b10-b7e6555fd628\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb" Oct 30 18:16:52.108030 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:52.107951 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb" Oct 30 18:16:52.416559 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:16:52.416524 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc18f5382_cae5_42b9_9b10_b7e6555fd628.slice/crio-df5e8740c59230543403777f37042aaad83dea172a6290950eee9c47a17cf692 WatchSource:0}: Error finding container df5e8740c59230543403777f37042aaad83dea172a6290950eee9c47a17cf692: Status 404 returned error can't find the container with id df5e8740c59230543403777f37042aaad83dea172a6290950eee9c47a17cf692 Oct 30 18:16:52.416763 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:52.416739 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb"] Oct 30 18:16:53.091101 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:53.091063 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd"] Oct 30 18:16:53.095678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:53.095657 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd" Oct 30 18:16:53.113852 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:53.113825 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd"] Oct 30 18:16:53.129925 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:53.129888 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb" event={"ID":"c18f5382-cae5-42b9-9b10-b7e6555fd628","Type":"ContainerStarted","Data":"d1d06a9301d39a9aff6833ac853ebe4bf35f0c4cb09b3e4189fc809b80475cdf"} Oct 30 18:16:53.130056 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:53.129952 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb" event={"ID":"c18f5382-cae5-42b9-9b10-b7e6555fd628","Type":"ContainerStarted","Data":"df5e8740c59230543403777f37042aaad83dea172a6290950eee9c47a17cf692"} Oct 30 18:16:53.174590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:53.174537 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb" podStartSLOduration=2.174521455 podStartE2EDuration="2.174521455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:53.17275272 +0000 UTC m=+912.489173040" watchObservedRunningTime="2026-10-30 18:16:53.174521455 +0000 UTC m=+912.490941767" Oct 30 18:16:53.185148 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:53.185110 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xhnx2\" (UniqueName: \"kubernetes.io/projected/72f09463-8af4-4ed2-b185-189878a28c11-kube-api-access-xhnx2\") pod \"maas-api-revoke-sub-5d67033e-b98kd\" (UID: \"72f09463-8af4-4ed2-b185-189878a28c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd" Oct 30 18:16:53.185428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:53.185411 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f09463-8af4-4ed2-b185-189878a28c11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d67033e-b98kd\" (UID: \"72f09463-8af4-4ed2-b185-189878a28c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd" Oct 30 18:16:53.185563 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:53.185549 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f09463-8af4-4ed2-b185-189878a28c11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d67033e-b98kd\" (UID: \"72f09463-8af4-4ed2-b185-189878a28c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd" Oct 30 18:16:53.286763 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:53.286734 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f09463-8af4-4ed2-b185-189878a28c11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d67033e-b98kd\" (UID: \"72f09463-8af4-4ed2-b185-189878a28c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd" Oct 30 18:16:53.286953 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:53.286834 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f09463-8af4-4ed2-b185-189878a28c11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d67033e-b98kd\" (UID: \"72f09463-8af4-4ed2-b185-189878a28c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd" Oct 30 18:16:53.286953 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:53.286864 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xhnx2\" (UniqueName: \"kubernetes.io/projected/72f09463-8af4-4ed2-b185-189878a28c11-kube-api-access-xhnx2\") pod \"maas-api-revoke-sub-5d67033e-b98kd\" (UID: \"72f09463-8af4-4ed2-b185-189878a28c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd" Oct 30 18:16:53.287403 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:53.287376 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f09463-8af4-4ed2-b185-189878a28c11-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5d67033e-b98kd\" (UID: \"72f09463-8af4-4ed2-b185-189878a28c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd" Oct 30 18:16:53.289374 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:53.289352 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f09463-8af4-4ed2-b185-189878a28c11-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5d67033e-b98kd\" (UID: \"72f09463-8af4-4ed2-b185-189878a28c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd" Oct 30 18:16:53.305778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:53.305754 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xhnx2\" (UniqueName: \"kubernetes.io/projected/72f09463-8af4-4ed2-b185-189878a28c11-kube-api-access-xhnx2\") pod \"maas-api-revoke-sub-5d67033e-b98kd\" (UID: \"72f09463-8af4-4ed2-b185-189878a28c11\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd" Oct 30 18:16:53.412273 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:53.412182 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd" Oct 30 18:16:53.717891 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:53.717852 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd"] Oct 30 18:16:53.718604 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:16:53.718569 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72f09463_8af4_4ed2_b185_189878a28c11.slice/crio-721ee7a1008824a3b738e85f0571a3145e24ae9bd3b7aaee74c41eda5894f661 WatchSource:0}: Error finding container 721ee7a1008824a3b738e85f0571a3145e24ae9bd3b7aaee74c41eda5894f661: Status 404 returned error can't find the container with id 721ee7a1008824a3b738e85f0571a3145e24ae9bd3b7aaee74c41eda5894f661 Oct 30 18:16:54.139472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:54.139429 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd" event={"ID":"72f09463-8af4-4ed2-b185-189878a28c11","Type":"ContainerStarted","Data":"0738da3cf01dbc44ff0bd79d3bd5904295e1190f6f4cf1d1e60fd38045bc5641"} Oct 30 18:16:54.139930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:54.139492 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd" event={"ID":"72f09463-8af4-4ed2-b185-189878a28c11","Type":"ContainerStarted","Data":"721ee7a1008824a3b738e85f0571a3145e24ae9bd3b7aaee74c41eda5894f661"} Oct 30 18:16:54.141177 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:54.141153 5232 generic.go:356] "Generic (PLEG): container finished" podID="c18f5382-cae5-42b9-9b10-b7e6555fd628" containerID="d1d06a9301d39a9aff6833ac853ebe4bf35f0c4cb09b3e4189fc809b80475cdf" exitCode=0 Oct 30 18:16:54.141302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:54.141227 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb" event={"ID":"c18f5382-cae5-42b9-9b10-b7e6555fd628","Type":"ContainerDied","Data":"d1d06a9301d39a9aff6833ac853ebe4bf35f0c4cb09b3e4189fc809b80475cdf"} Oct 30 18:16:54.180917 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:54.180857 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd" podStartSLOduration=1.180843194 podStartE2EDuration="1.180843194s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:16:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:16:54.176434741 +0000 UTC m=+913.492855063" watchObservedRunningTime="2026-10-30 18:16:54.180843194 +0000 UTC m=+913.497263515" Oct 30 18:16:55.150467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:55.150430 5232 generic.go:356] "Generic (PLEG): container finished" podID="72f09463-8af4-4ed2-b185-189878a28c11" containerID="0738da3cf01dbc44ff0bd79d3bd5904295e1190f6f4cf1d1e60fd38045bc5641" exitCode=0 Oct 30 18:16:55.150878 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:55.150545 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd" event={"ID":"72f09463-8af4-4ed2-b185-189878a28c11","Type":"ContainerDied","Data":"0738da3cf01dbc44ff0bd79d3bd5904295e1190f6f4cf1d1e60fd38045bc5641"} Oct 30 18:16:55.324354 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:55.324333 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb" Oct 30 18:16:55.425698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:55.425622 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c18f5382-cae5-42b9-9b10-b7e6555fd628-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18f5382-cae5-42b9-9b10-b7e6555fd628-maas-api-xks-ca\") pod \"c18f5382-cae5-42b9-9b10-b7e6555fd628\" (UID: \"c18f5382-cae5-42b9-9b10-b7e6555fd628\") " Oct 30 18:16:55.425698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:55.425665 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c18f5382-cae5-42b9-9b10-b7e6555fd628-kube-api-access-6tsj2\" (UniqueName: \"kubernetes.io/projected/c18f5382-cae5-42b9-9b10-b7e6555fd628-kube-api-access-6tsj2\") pod \"c18f5382-cae5-42b9-9b10-b7e6555fd628\" (UID: \"c18f5382-cae5-42b9-9b10-b7e6555fd628\") " Oct 30 18:16:55.425936 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:55.425720 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c18f5382-cae5-42b9-9b10-b7e6555fd628-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18f5382-cae5-42b9-9b10-b7e6555fd628-maas-api-openshift-ca\") pod \"c18f5382-cae5-42b9-9b10-b7e6555fd628\" (UID: \"c18f5382-cae5-42b9-9b10-b7e6555fd628\") " Oct 30 18:16:55.426210 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:55.426180 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c18f5382-cae5-42b9-9b10-b7e6555fd628-maas-api-openshift-ca" pod "c18f5382-cae5-42b9-9b10-b7e6555fd628" (UID: "c18f5382-cae5-42b9-9b10-b7e6555fd628"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:55.426358 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:55.426336 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c18f5382-cae5-42b9-9b10-b7e6555fd628-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:55.428114 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:55.428087 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c18f5382-cae5-42b9-9b10-b7e6555fd628-maas-api-xks-ca" pod "c18f5382-cae5-42b9-9b10-b7e6555fd628" (UID: "c18f5382-cae5-42b9-9b10-b7e6555fd628"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:55.428203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:55.428135 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c18f5382-cae5-42b9-9b10-b7e6555fd628-kube-api-access-6tsj2" pod "c18f5382-cae5-42b9-9b10-b7e6555fd628" (UID: "c18f5382-cae5-42b9-9b10-b7e6555fd628"). InnerVolumeSpecName "kube-api-access-6tsj2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:55.532534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:55.530635 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c18f5382-cae5-42b9-9b10-b7e6555fd628-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:55.532534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:55.530677 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6tsj2\" (UniqueName: \"kubernetes.io/projected/c18f5382-cae5-42b9-9b10-b7e6555fd628-kube-api-access-6tsj2\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:56.512584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:56.506114 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb" Oct 30 18:16:56.512584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:56.511933 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb" event={"ID":"c18f5382-cae5-42b9-9b10-b7e6555fd628","Type":"ContainerDied","Data":"df5e8740c59230543403777f37042aaad83dea172a6290950eee9c47a17cf692"} Oct 30 18:16:56.512584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:56.512065 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df5e8740c59230543403777f37042aaad83dea172a6290950eee9c47a17cf692" Oct 30 18:16:57.711578 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:16:57.710746 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:16:57.725539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:57.721737 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd" Oct 30 18:16:57.817585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:57.812622 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:57.817585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:57.814475 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72f09463-8af4-4ed2-b185-189878a28c11-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f09463-8af4-4ed2-b185-189878a28c11-maas-api-xks-ca\") pod \"72f09463-8af4-4ed2-b185-189878a28c11\" (UID: \"72f09463-8af4-4ed2-b185-189878a28c11\") " Oct 30 18:16:57.817585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:57.814656 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72f09463-8af4-4ed2-b185-189878a28c11-kube-api-access-xhnx2\" (UniqueName: \"kubernetes.io/projected/72f09463-8af4-4ed2-b185-189878a28c11-kube-api-access-xhnx2\") pod \"72f09463-8af4-4ed2-b185-189878a28c11\" (UID: \"72f09463-8af4-4ed2-b185-189878a28c11\") " Oct 30 18:16:57.817585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:57.814739 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72f09463-8af4-4ed2-b185-189878a28c11-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f09463-8af4-4ed2-b185-189878a28c11-maas-api-openshift-ca\") pod \"72f09463-8af4-4ed2-b185-189878a28c11\" (UID: \"72f09463-8af4-4ed2-b185-189878a28c11\") " Oct 30 18:16:57.819721 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:57.819682 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72f09463-8af4-4ed2-b185-189878a28c11-maas-api-openshift-ca" pod "72f09463-8af4-4ed2-b185-189878a28c11" (UID: "72f09463-8af4-4ed2-b185-189878a28c11"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:16:57.844104 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:57.844063 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72f09463-8af4-4ed2-b185-189878a28c11-kube-api-access-xhnx2" pod "72f09463-8af4-4ed2-b185-189878a28c11" (UID: "72f09463-8af4-4ed2-b185-189878a28c11"). InnerVolumeSpecName "kube-api-access-xhnx2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:16:57.848594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:57.844412 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72f09463-8af4-4ed2-b185-189878a28c11-maas-api-xks-ca" pod "72f09463-8af4-4ed2-b185-189878a28c11" (UID: "72f09463-8af4-4ed2-b185-189878a28c11"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:16:57.944536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:57.917014 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhnx2\" (UniqueName: \"kubernetes.io/projected/72f09463-8af4-4ed2-b185-189878a28c11-kube-api-access-xhnx2\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:57.944536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:57.917052 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72f09463-8af4-4ed2-b185-189878a28c11-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:57.944536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:57.917067 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72f09463-8af4-4ed2-b185-189878a28c11-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:16:58.378338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:58.377606 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.405535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:58.402818 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.405535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:58.404128 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.405535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:58.404424 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:16:58.405535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:58.404740 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:16:58.481015 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:58.479148 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.505057 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:58.505014 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.629496 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:58.629405 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd" event={"ID":"72f09463-8af4-4ed2-b185-189878a28c11","Type":"ContainerDied","Data":"721ee7a1008824a3b738e85f0571a3145e24ae9bd3b7aaee74c41eda5894f661"} Oct 30 18:16:58.629752 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:58.629732 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="721ee7a1008824a3b738e85f0571a3145e24ae9bd3b7aaee74c41eda5894f661" Oct 30 18:16:58.629923 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:58.629909 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd" Oct 30 18:16:58.647644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:58.647598 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.722684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:58.722636 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.747548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:58.746543 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.775558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:58.765190 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.823033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:58.822983 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:58.888604 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:58.879731 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:16:59.321827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.318817 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": context deadline exceeded" Oct 30 18:16:59.442572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.417614 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.442572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.418106 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:16:59.442572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.418390 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:16:59.442572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.418658 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:16:59.442572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.418952 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:16:59.442572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.419011 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:16:59.442572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.419048 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:16:59.442572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.419081 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:16:59.491578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.443196 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.491578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.457579 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:16:59.532534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.521884 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:16:59.597389 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.547769 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.655240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.587406 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:16:59.655556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.655525 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:16:59.655753 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.655738 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:16:59.810534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.793731 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:16:59.834988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.834943 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.836146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.836112 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:16:59.889912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:16:59.889819 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.001223 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:00.001183 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.063374 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:00.063328 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.068626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:00.068585 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.120566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:00.098804 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:17:00.120566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:00.100108 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:17:00.120566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:00.100206 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:17:00.450576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:00.429858 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.450576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:00.441757 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.479575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:00.452162 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.479575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:00.458576 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:17:00.684533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:00.600127 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:17:00.684533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:00.654611 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.684533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:00.655089 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:00.881658 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:00.881608 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:01.228054 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:01.227930 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:01.398687 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:01.228527 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:17:01.400923 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:01.228752 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:17:01.445501 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:01.445434 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:01.950939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:01.473464 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:01.952900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:01.228934 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Oct 30 18:17:01.953119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:01.546239 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:17:01.993711 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:01.563577 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:17:01.995111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:01.661902 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:01.996339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:01.668546 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:17:01.997431 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:01.669031 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:02.092625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:02.092584 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:17:02.422784 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:02.422735 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:02.495738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:02.495693 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:17:02.496547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:02.496493 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:17:02.867685 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:02.867642 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:17:02.867976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:02.867934 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" podUID="f9270d40-a0b1-4c84-a28b-9f4bca0487be" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.49:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:02.903575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:02.884722 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:02.955774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:02.955726 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:02.969970 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:02.969911 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:17:03.004628 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:02.991764 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:03.249595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.224793 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:03.303485 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.290119 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 18:17:03.303485 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.290362 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:17:03.303485 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.290541 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:17:03.327536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.311014 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:17:03.327536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.311091 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" podUID="cc39faea-f977-4ba2-be13-cef341667507" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.77:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:03.352556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.351095 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:17:03.352556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.351196 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" podUID="5d5ba31c-c8f3-4467-be05-fd146702503e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:03.372533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.372088 5232 patch_prober.go:29] interesting pod/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:17:03.372533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.372157 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" podUID="c7a2a719-e154-4072-93f2-ed366344dbbc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:03.525588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.481290 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:17:03.572365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.507932 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:17:03.572365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.509181 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:17:03.572365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.515815 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:17:03.572365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.566010 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" podUID="a337e101-6a41-4416-9731-f28cf2afd0f2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.50:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:03.572365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.559004 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:17:03.572365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.566067 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" podUID="03bc2ff1-ddb6-41cf-8238-4df762a46c0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:03.613806 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:17:03.592612 5232 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="2.73s" Oct 30 18:17:03.667001 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.663857 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:17:03.667001 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.663916 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" podUID="8a806882-2d84-427c-8698-9ed54994b748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:03.901586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.892713 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:03.956070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.956033 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx"] Oct 30 18:17:03.971192 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.971158 5232 cpu_manager.go:414] "containerMap: removing container" podUID="c18f5382-cae5-42b9-9b10-b7e6555fd628" containerName="revoke-keys" Oct 30 18:17:03.983836 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.983801 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c18f5382-cae5-42b9-9b10-b7e6555fd628" containerName="revoke-keys" Oct 30 18:17:03.984151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.984134 5232 cpu_manager.go:414] "containerMap: removing container" podUID="72f09463-8af4-4ed2-b185-189878a28c11" containerName="revoke-keys" Oct 30 18:17:03.985531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.985486 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72f09463-8af4-4ed2-b185-189878a28c11" containerName="revoke-keys" Oct 30 18:17:03.985880 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.976637 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:17:03.987303 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.987196 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="c18f5382-cae5-42b9-9b10-b7e6555fd628" containerName="revoke-keys" Oct 30 18:17:03.987446 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.987434 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="72f09463-8af4-4ed2-b185-189878a28c11" containerName="revoke-keys" Oct 30 18:17:03.993645 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.993612 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:03.997028 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.996443 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:17:03.997299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:03.996573 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx" Oct 30 18:17:04.001195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.001172 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx"] Oct 30 18:17:04.012953 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.012923 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:17:04.016472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.014393 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:17:04.104120 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.102709 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:17:04.200886 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.190324 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672af468-d0e0-4276-b532-add124f8b2a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd0cd224-8z6cx\" (UID: \"672af468-d0e0-4276-b532-add124f8b2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx" Oct 30 18:17:04.201285 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.201258 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82mgh\" (UniqueName: \"kubernetes.io/projected/672af468-d0e0-4276-b532-add124f8b2a5-kube-api-access-82mgh\") pod \"maas-api-revoke-sub-fd0cd224-8z6cx\" (UID: \"672af468-d0e0-4276-b532-add124f8b2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx" Oct 30 18:17:04.201557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.201527 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672af468-d0e0-4276-b532-add124f8b2a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd0cd224-8z6cx\" (UID: \"672af468-d0e0-4276-b532-add124f8b2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx" Oct 30 18:17:04.302237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.290614 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:04.302237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.290780 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:04.302237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.291126 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:04.302237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.291414 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:04.316531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.308315 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82mgh\" (UniqueName: \"kubernetes.io/projected/672af468-d0e0-4276-b532-add124f8b2a5-kube-api-access-82mgh\") pod \"maas-api-revoke-sub-fd0cd224-8z6cx\" (UID: \"672af468-d0e0-4276-b532-add124f8b2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx" Oct 30 18:17:04.317258 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.317230 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672af468-d0e0-4276-b532-add124f8b2a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd0cd224-8z6cx\" (UID: \"672af468-d0e0-4276-b532-add124f8b2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx" Oct 30 18:17:04.317487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.317471 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672af468-d0e0-4276-b532-add124f8b2a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd0cd224-8z6cx\" (UID: \"672af468-d0e0-4276-b532-add124f8b2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx" Oct 30 18:17:04.318566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.318542 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672af468-d0e0-4276-b532-add124f8b2a5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fd0cd224-8z6cx\" (UID: \"672af468-d0e0-4276-b532-add124f8b2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx" Oct 30 18:17:04.380169 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.355219 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82mgh\" (UniqueName: \"kubernetes.io/projected/672af468-d0e0-4276-b532-add124f8b2a5-kube-api-access-82mgh\") pod \"maas-api-revoke-sub-fd0cd224-8z6cx\" (UID: \"672af468-d0e0-4276-b532-add124f8b2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx" Oct 30 18:17:04.386338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.386304 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672af468-d0e0-4276-b532-add124f8b2a5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fd0cd224-8z6cx\" (UID: \"672af468-d0e0-4276-b532-add124f8b2a5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx" Oct 30 18:17:04.391830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.391806 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx" Oct 30 18:17:04.398669 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.396053 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:17:04.435666 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.434180 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:17:04.438145 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.437612 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:17:04.449282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.447744 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:17:04.456442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.456163 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:17:04.479537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.473759 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:17:04.481796 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.481763 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:17:04.522702 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.521630 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:17:04.523036 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.523014 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:17:04.796266 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:17:04.796236 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672af468_d0e0_4276_b532_add124f8b2a5.slice/crio-283d608d3a9aaee497d21b78ce933de2db361a1b30e044cffec8a426c943eb0d WatchSource:0}: Error finding container 283d608d3a9aaee497d21b78ce933de2db361a1b30e044cffec8a426c943eb0d: Status 404 returned error can't find the container with id 283d608d3a9aaee497d21b78ce933de2db361a1b30e044cffec8a426c943eb0d Oct 30 18:17:04.809300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:04.809273 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx"] Oct 30 18:17:05.404977 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:05.401599 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g"] Oct 30 18:17:05.414806 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:05.414777 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g" Oct 30 18:17:05.419913 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:05.417490 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g"] Oct 30 18:17:05.570068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:05.570024 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49rgm\" (UniqueName: \"kubernetes.io/projected/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-kube-api-access-49rgm\") pod \"maas-api-revoke-sub-28f0a739-jbb8g\" (UID: \"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g" Oct 30 18:17:05.570222 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:05.570136 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28f0a739-jbb8g\" (UID: \"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g" Oct 30 18:17:05.570323 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:05.570308 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28f0a739-jbb8g\" (UID: \"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g" Oct 30 18:17:05.587767 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:05.587710 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx" event={"ID":"672af468-d0e0-4276-b532-add124f8b2a5","Type":"ContainerStarted","Data":"a4394484a4fc36dc9bdb68c258090b3c628f4ac38b3e80e50f96bb5451abaf4d"} Oct 30 18:17:05.587960 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:05.587828 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx" event={"ID":"672af468-d0e0-4276-b532-add124f8b2a5","Type":"ContainerStarted","Data":"283d608d3a9aaee497d21b78ce933de2db361a1b30e044cffec8a426c943eb0d"} Oct 30 18:17:05.627729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:05.627677 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx" podStartSLOduration=3.627662976 podStartE2EDuration="3.627662976s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:05.625833036 +0000 UTC m=+924.942253358" watchObservedRunningTime="2026-10-30 18:17:05.627662976 +0000 UTC m=+924.944083328" Oct 30 18:17:05.671303 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:05.671224 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28f0a739-jbb8g\" (UID: \"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g" Oct 30 18:17:05.671434 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:05.671374 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49rgm\" (UniqueName: \"kubernetes.io/projected/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-kube-api-access-49rgm\") pod \"maas-api-revoke-sub-28f0a739-jbb8g\" (UID: \"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g" Oct 30 18:17:05.671434 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:05.671418 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28f0a739-jbb8g\" (UID: \"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g" Oct 30 18:17:05.671901 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:05.671877 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28f0a739-jbb8g\" (UID: \"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g" Oct 30 18:17:05.674006 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:05.673980 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28f0a739-jbb8g\" (UID: \"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g" Oct 30 18:17:05.690282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:05.690254 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49rgm\" (UniqueName: \"kubernetes.io/projected/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-kube-api-access-49rgm\") pod \"maas-api-revoke-sub-28f0a739-jbb8g\" (UID: \"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g" Oct 30 18:17:05.811836 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:05.811809 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g" Oct 30 18:17:06.114211 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.114182 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g"] Oct 30 18:17:06.114962 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:17:06.114925 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbeb8bf6d_8c9f_43d0_8a35_ea3ab5c24c95.slice/crio-26d6c8bdfd024793d586dff69c84cf3bd31326ac04b57e82c3e491b97f5f25a3 WatchSource:0}: Error finding container 26d6c8bdfd024793d586dff69c84cf3bd31326ac04b57e82c3e491b97f5f25a3: Status 404 returned error can't find the container with id 26d6c8bdfd024793d586dff69c84cf3bd31326ac04b57e82c3e491b97f5f25a3 Oct 30 18:17:06.223807 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.223774 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9"] Oct 30 18:17:06.229744 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.229721 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9" Oct 30 18:17:06.245877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.245848 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9"] Oct 30 18:17:06.382635 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.382555 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d2382d-51b2-4f31-aa8e-05205da06225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c105a403-6cks9\" (UID: \"39d2382d-51b2-4f31-aa8e-05205da06225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9" Oct 30 18:17:06.382635 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.382612 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d2382d-51b2-4f31-aa8e-05205da06225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c105a403-6cks9\" (UID: \"39d2382d-51b2-4f31-aa8e-05205da06225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9" Oct 30 18:17:06.382827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.382775 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tbxpx\" (UniqueName: \"kubernetes.io/projected/39d2382d-51b2-4f31-aa8e-05205da06225-kube-api-access-tbxpx\") pod \"maas-api-revoke-sub-c105a403-6cks9\" (UID: \"39d2382d-51b2-4f31-aa8e-05205da06225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9" Oct 30 18:17:06.485017 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.484984 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d2382d-51b2-4f31-aa8e-05205da06225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c105a403-6cks9\" (UID: \"39d2382d-51b2-4f31-aa8e-05205da06225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9" Oct 30 18:17:06.485450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.485053 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d2382d-51b2-4f31-aa8e-05205da06225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c105a403-6cks9\" (UID: \"39d2382d-51b2-4f31-aa8e-05205da06225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9" Oct 30 18:17:06.485450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.485165 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tbxpx\" (UniqueName: \"kubernetes.io/projected/39d2382d-51b2-4f31-aa8e-05205da06225-kube-api-access-tbxpx\") pod \"maas-api-revoke-sub-c105a403-6cks9\" (UID: \"39d2382d-51b2-4f31-aa8e-05205da06225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9" Oct 30 18:17:06.485665 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.485644 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d2382d-51b2-4f31-aa8e-05205da06225-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c105a403-6cks9\" (UID: \"39d2382d-51b2-4f31-aa8e-05205da06225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9" Oct 30 18:17:06.487475 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.487455 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d2382d-51b2-4f31-aa8e-05205da06225-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c105a403-6cks9\" (UID: \"39d2382d-51b2-4f31-aa8e-05205da06225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9" Oct 30 18:17:06.504320 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.504294 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tbxpx\" (UniqueName: \"kubernetes.io/projected/39d2382d-51b2-4f31-aa8e-05205da06225-kube-api-access-tbxpx\") pod \"maas-api-revoke-sub-c105a403-6cks9\" (UID: \"39d2382d-51b2-4f31-aa8e-05205da06225\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9" Oct 30 18:17:06.572534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.561889 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9" Oct 30 18:17:06.682622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.681474 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx" event={"ID":"672af468-d0e0-4276-b532-add124f8b2a5","Type":"ContainerDied","Data":"a4394484a4fc36dc9bdb68c258090b3c628f4ac38b3e80e50f96bb5451abaf4d"} Oct 30 18:17:06.682622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.662812 5232 generic.go:356] "Generic (PLEG): container finished" podID="672af468-d0e0-4276-b532-add124f8b2a5" containerID="a4394484a4fc36dc9bdb68c258090b3c628f4ac38b3e80e50f96bb5451abaf4d" exitCode=0 Oct 30 18:17:06.684464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.684435 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g" event={"ID":"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95","Type":"ContainerStarted","Data":"5893347a5d0f62fb0eebec7d6b32e43a72ef938cd9f1e65cd701bf06b65e6a17"} Oct 30 18:17:06.684631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.684479 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g" event={"ID":"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95","Type":"ContainerStarted","Data":"26d6c8bdfd024793d586dff69c84cf3bd31326ac04b57e82c3e491b97f5f25a3"} Oct 30 18:17:06.774536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:06.773631 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g" podStartSLOduration=1.7736085240000001 podStartE2EDuration="1.773608524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:06.767390087 +0000 UTC m=+926.083810407" watchObservedRunningTime="2026-10-30 18:17:06.773608524 +0000 UTC m=+926.090028849" Oct 30 18:17:07.001274 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:07.001248 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9"] Oct 30 18:17:07.001824 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:17:07.001794 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39d2382d_51b2_4f31_aa8e_05205da06225.slice/crio-d498468979083feb002bbdfcd67880a3f3a6426745dc6f7bffaa255145678481 WatchSource:0}: Error finding container d498468979083feb002bbdfcd67880a3f3a6426745dc6f7bffaa255145678481: Status 404 returned error can't find the container with id d498468979083feb002bbdfcd67880a3f3a6426745dc6f7bffaa255145678481 Oct 30 18:17:07.695199 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:07.695100 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9" event={"ID":"39d2382d-51b2-4f31-aa8e-05205da06225","Type":"ContainerStarted","Data":"26cc352eaf761b48087a2b1c5a0c6441ed7a7fc806dd6690436f824266523ca9"} Oct 30 18:17:07.695199 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:07.695159 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9" event={"ID":"39d2382d-51b2-4f31-aa8e-05205da06225","Type":"ContainerStarted","Data":"d498468979083feb002bbdfcd67880a3f3a6426745dc6f7bffaa255145678481"} Oct 30 18:17:07.696783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:07.696757 5232 generic.go:356] "Generic (PLEG): container finished" podID="beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95" containerID="5893347a5d0f62fb0eebec7d6b32e43a72ef938cd9f1e65cd701bf06b65e6a17" exitCode=0 Oct 30 18:17:07.696915 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:07.696830 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g" event={"ID":"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95","Type":"ContainerDied","Data":"5893347a5d0f62fb0eebec7d6b32e43a72ef938cd9f1e65cd701bf06b65e6a17"} Oct 30 18:17:07.734717 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:07.734665 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9" podStartSLOduration=1.734651403 podStartE2EDuration="1.734651403s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:07.732556146 +0000 UTC m=+927.048976468" watchObservedRunningTime="2026-10-30 18:17:07.734651403 +0000 UTC m=+927.051071723" Oct 30 18:17:07.865635 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:07.865609 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx" Oct 30 18:17:07.909559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:07.909525 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/672af468-d0e0-4276-b532-add124f8b2a5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672af468-d0e0-4276-b532-add124f8b2a5-maas-api-openshift-ca\") pod \"672af468-d0e0-4276-b532-add124f8b2a5\" (UID: \"672af468-d0e0-4276-b532-add124f8b2a5\") " Oct 30 18:17:07.909726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:07.909577 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/672af468-d0e0-4276-b532-add124f8b2a5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672af468-d0e0-4276-b532-add124f8b2a5-maas-api-xks-ca\") pod \"672af468-d0e0-4276-b532-add124f8b2a5\" (UID: \"672af468-d0e0-4276-b532-add124f8b2a5\") " Oct 30 18:17:07.909726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:07.909638 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/672af468-d0e0-4276-b532-add124f8b2a5-kube-api-access-82mgh\" (UniqueName: \"kubernetes.io/projected/672af468-d0e0-4276-b532-add124f8b2a5-kube-api-access-82mgh\") pod \"672af468-d0e0-4276-b532-add124f8b2a5\" (UID: \"672af468-d0e0-4276-b532-add124f8b2a5\") " Oct 30 18:17:07.909905 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:07.909871 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/672af468-d0e0-4276-b532-add124f8b2a5-maas-api-openshift-ca" pod "672af468-d0e0-4276-b532-add124f8b2a5" (UID: "672af468-d0e0-4276-b532-add124f8b2a5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:07.910756 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:07.910725 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/672af468-d0e0-4276-b532-add124f8b2a5-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:07.912214 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:07.912184 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/672af468-d0e0-4276-b532-add124f8b2a5-kube-api-access-82mgh" pod "672af468-d0e0-4276-b532-add124f8b2a5" (UID: "672af468-d0e0-4276-b532-add124f8b2a5"). InnerVolumeSpecName "kube-api-access-82mgh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:07.912434 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:07.912406 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/672af468-d0e0-4276-b532-add124f8b2a5-maas-api-xks-ca" pod "672af468-d0e0-4276-b532-add124f8b2a5" (UID: "672af468-d0e0-4276-b532-add124f8b2a5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:08.012289 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.012251 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/672af468-d0e0-4276-b532-add124f8b2a5-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:08.012289 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.012283 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82mgh\" (UniqueName: \"kubernetes.io/projected/672af468-d0e0-4276-b532-add124f8b2a5-kube-api-access-82mgh\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:08.304538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.304393 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7"] Oct 30 18:17:08.305988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.305961 5232 cpu_manager.go:414] "containerMap: removing container" podUID="672af468-d0e0-4276-b532-add124f8b2a5" containerName="revoke-keys" Oct 30 18:17:08.306186 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.306170 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="672af468-d0e0-4276-b532-add124f8b2a5" containerName="revoke-keys" Oct 30 18:17:08.306494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.306479 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="672af468-d0e0-4276-b532-add124f8b2a5" containerName="revoke-keys" Oct 30 18:17:08.393417 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.393383 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7"] Oct 30 18:17:08.393664 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.393547 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7" Oct 30 18:17:08.423814 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.423776 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-82pcv\" (UniqueName: \"kubernetes.io/projected/e0b82ac4-2aae-47c4-b032-ab5e463a7920-kube-api-access-82pcv\") pod \"maas-api-revoke-sub-a904b6fd-59kj7\" (UID: \"e0b82ac4-2aae-47c4-b032-ab5e463a7920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7" Oct 30 18:17:08.423977 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.423960 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b82ac4-2aae-47c4-b032-ab5e463a7920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a904b6fd-59kj7\" (UID: \"e0b82ac4-2aae-47c4-b032-ab5e463a7920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7" Oct 30 18:17:08.424098 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.424078 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b82ac4-2aae-47c4-b032-ab5e463a7920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a904b6fd-59kj7\" (UID: \"e0b82ac4-2aae-47c4-b032-ab5e463a7920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7" Oct 30 18:17:08.525032 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.524999 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b82ac4-2aae-47c4-b032-ab5e463a7920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a904b6fd-59kj7\" (UID: \"e0b82ac4-2aae-47c4-b032-ab5e463a7920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7" Oct 30 18:17:08.525207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.525186 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-82pcv\" (UniqueName: \"kubernetes.io/projected/e0b82ac4-2aae-47c4-b032-ab5e463a7920-kube-api-access-82pcv\") pod \"maas-api-revoke-sub-a904b6fd-59kj7\" (UID: \"e0b82ac4-2aae-47c4-b032-ab5e463a7920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7" Oct 30 18:17:08.525358 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.525341 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b82ac4-2aae-47c4-b032-ab5e463a7920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a904b6fd-59kj7\" (UID: \"e0b82ac4-2aae-47c4-b032-ab5e463a7920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7" Oct 30 18:17:08.526012 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.525992 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b82ac4-2aae-47c4-b032-ab5e463a7920-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a904b6fd-59kj7\" (UID: \"e0b82ac4-2aae-47c4-b032-ab5e463a7920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7" Oct 30 18:17:08.527636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.527611 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b82ac4-2aae-47c4-b032-ab5e463a7920-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a904b6fd-59kj7\" (UID: \"e0b82ac4-2aae-47c4-b032-ab5e463a7920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7" Oct 30 18:17:08.545011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.544991 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-82pcv\" (UniqueName: \"kubernetes.io/projected/e0b82ac4-2aae-47c4-b032-ab5e463a7920-kube-api-access-82pcv\") pod \"maas-api-revoke-sub-a904b6fd-59kj7\" (UID: \"e0b82ac4-2aae-47c4-b032-ab5e463a7920\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7" Oct 30 18:17:08.705480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.705403 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx" Oct 30 18:17:08.705863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.705404 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx" event={"ID":"672af468-d0e0-4276-b532-add124f8b2a5","Type":"ContainerDied","Data":"283d608d3a9aaee497d21b78ce933de2db361a1b30e044cffec8a426c943eb0d"} Oct 30 18:17:08.705863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.705571 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="283d608d3a9aaee497d21b78ce933de2db361a1b30e044cffec8a426c943eb0d" Oct 30 18:17:08.707239 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.707214 5232 generic.go:356] "Generic (PLEG): container finished" podID="39d2382d-51b2-4f31-aa8e-05205da06225" containerID="26cc352eaf761b48087a2b1c5a0c6441ed7a7fc806dd6690436f824266523ca9" exitCode=0 Oct 30 18:17:08.707367 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.707299 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9" event={"ID":"39d2382d-51b2-4f31-aa8e-05205da06225","Type":"ContainerDied","Data":"26cc352eaf761b48087a2b1c5a0c6441ed7a7fc806dd6690436f824266523ca9"} Oct 30 18:17:08.714237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.714218 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7" Oct 30 18:17:08.896230 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.896203 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g" Oct 30 18:17:08.915395 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.915351 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn"] Oct 30 18:17:08.916850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.916825 5232 cpu_manager.go:414] "containerMap: removing container" podUID="beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95" containerName="revoke-keys" Oct 30 18:17:08.916850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.916849 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95" containerName="revoke-keys" Oct 30 18:17:08.917286 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.917266 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95" containerName="revoke-keys" Oct 30 18:17:08.922657 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.922637 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn" Oct 30 18:17:08.930935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.930762 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-maas-api-openshift-ca\") pod \"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95\" (UID: \"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95\") " Oct 30 18:17:08.930935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.930805 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-kube-api-access-49rgm\" (UniqueName: \"kubernetes.io/projected/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-kube-api-access-49rgm\") pod \"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95\" (UID: \"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95\") " Oct 30 18:17:08.931133 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.931113 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-maas-api-xks-ca\") pod \"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95\" (UID: \"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95\") " Oct 30 18:17:08.931369 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.931351 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cb79ad-89d4-498e-be40-10ee94aa6861-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c6a9955-vwczn\" (UID: \"81cb79ad-89d4-498e-be40-10ee94aa6861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn" Oct 30 18:17:08.931442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.931422 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cb79ad-89d4-498e-be40-10ee94aa6861-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c6a9955-vwczn\" (UID: \"81cb79ad-89d4-498e-be40-10ee94aa6861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn" Oct 30 18:17:08.931704 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.931681 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dncz\" (UniqueName: \"kubernetes.io/projected/81cb79ad-89d4-498e-be40-10ee94aa6861-kube-api-access-4dncz\") pod \"maas-api-revoke-sub-0c6a9955-vwczn\" (UID: \"81cb79ad-89d4-498e-be40-10ee94aa6861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn" Oct 30 18:17:08.931863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.931836 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-maas-api-openshift-ca" pod "beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95" (UID: "beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:08.936490 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.936459 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn"] Oct 30 18:17:08.939686 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.939662 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-maas-api-xks-ca" pod "beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95" (UID: "beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:08.949713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:08.949677 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-kube-api-access-49rgm" pod "beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95" (UID: "beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95"). InnerVolumeSpecName "kube-api-access-49rgm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:09.028859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.028829 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7"] Oct 30 18:17:09.029171 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:17:09.029124 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0b82ac4_2aae_47c4_b032_ab5e463a7920.slice/crio-bd09a8b32a482f3a579f8d0376bf4b230e4925c86dbd9e2c798f9b64d656fc46 WatchSource:0}: Error finding container bd09a8b32a482f3a579f8d0376bf4b230e4925c86dbd9e2c798f9b64d656fc46: Status 404 returned error can't find the container with id bd09a8b32a482f3a579f8d0376bf4b230e4925c86dbd9e2c798f9b64d656fc46 Oct 30 18:17:09.032447 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.032420 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dncz\" (UniqueName: \"kubernetes.io/projected/81cb79ad-89d4-498e-be40-10ee94aa6861-kube-api-access-4dncz\") pod \"maas-api-revoke-sub-0c6a9955-vwczn\" (UID: \"81cb79ad-89d4-498e-be40-10ee94aa6861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn" Oct 30 18:17:09.032644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.032562 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cb79ad-89d4-498e-be40-10ee94aa6861-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c6a9955-vwczn\" (UID: \"81cb79ad-89d4-498e-be40-10ee94aa6861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn" Oct 30 18:17:09.032644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.032629 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cb79ad-89d4-498e-be40-10ee94aa6861-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c6a9955-vwczn\" (UID: \"81cb79ad-89d4-498e-be40-10ee94aa6861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn" Oct 30 18:17:09.032859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.032839 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:09.032980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.032864 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:09.032980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.032879 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-49rgm\" (UniqueName: \"kubernetes.io/projected/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95-kube-api-access-49rgm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:09.033289 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.033268 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cb79ad-89d4-498e-be40-10ee94aa6861-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0c6a9955-vwczn\" (UID: \"81cb79ad-89d4-498e-be40-10ee94aa6861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn" Oct 30 18:17:09.035245 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.035224 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cb79ad-89d4-498e-be40-10ee94aa6861-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0c6a9955-vwczn\" (UID: \"81cb79ad-89d4-498e-be40-10ee94aa6861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn" Oct 30 18:17:09.056523 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.056486 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dncz\" (UniqueName: \"kubernetes.io/projected/81cb79ad-89d4-498e-be40-10ee94aa6861-kube-api-access-4dncz\") pod \"maas-api-revoke-sub-0c6a9955-vwczn\" (UID: \"81cb79ad-89d4-498e-be40-10ee94aa6861\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn" Oct 30 18:17:09.070762 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.070734 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527"] Oct 30 18:17:09.076949 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.076929 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.095609 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.095088 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527"] Oct 30 18:17:09.139001 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.138951 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.139203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.139064 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.139203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.139113 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2hzwp\" (UniqueName: \"kubernetes.io/projected/c5923f01-3342-409d-8cc6-49e03be3a4ab-kube-api-access-2hzwp\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.139349 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.139228 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.139349 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.139271 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.139461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.139385 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.139461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.139432 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5923f01-3342-409d-8cc6-49e03be3a4ab-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.139650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.139465 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.139650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.139608 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.239853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.239823 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn" Oct 30 18:17:09.241122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.241087 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.241251 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.241154 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2hzwp\" (UniqueName: \"kubernetes.io/projected/c5923f01-3342-409d-8cc6-49e03be3a4ab-kube-api-access-2hzwp\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.241251 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.241215 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.241353 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.241251 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.241406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.241367 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.241471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.241412 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5923f01-3342-409d-8cc6-49e03be3a4ab-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.241541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.241471 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.241541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.241486 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.241652 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.241556 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.241652 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.241626 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.241759 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.241708 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.242049 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.242029 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.242166 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.242030 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.242372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.242350 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5923f01-3342-409d-8cc6-49e03be3a4ab-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.244002 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.243976 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.244174 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.244158 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.263418 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.263393 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.263776 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.263755 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2hzwp\" (UniqueName: \"kubernetes.io/projected/c5923f01-3342-409d-8cc6-49e03be3a4ab-kube-api-access-2hzwp\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.393012 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.392980 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:09.549063 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.549028 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn"] Oct 30 18:17:09.709749 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.709723 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527"] Oct 30 18:17:09.713279 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:17:09.713244 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5923f01_3342_409d_8cc6_49e03be3a4ab.slice/crio-7ac81f87e7d7fa0d084d4babd0ee537328ee36c8932ad305ab18252dcfe078fe WatchSource:0}: Error finding container 7ac81f87e7d7fa0d084d4babd0ee537328ee36c8932ad305ab18252dcfe078fe: Status 404 returned error can't find the container with id 7ac81f87e7d7fa0d084d4babd0ee537328ee36c8932ad305ab18252dcfe078fe Oct 30 18:17:09.716087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.716052 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:17:09.716178 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.716124 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:17:09.716178 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.716158 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:17:09.718469 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.718443 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g" event={"ID":"beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95","Type":"ContainerDied","Data":"26d6c8bdfd024793d586dff69c84cf3bd31326ac04b57e82c3e491b97f5f25a3"} Oct 30 18:17:09.718572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.718453 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g" Oct 30 18:17:09.718572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.718499 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="26d6c8bdfd024793d586dff69c84cf3bd31326ac04b57e82c3e491b97f5f25a3" Oct 30 18:17:09.720281 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.720258 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7" event={"ID":"e0b82ac4-2aae-47c4-b032-ab5e463a7920","Type":"ContainerStarted","Data":"9362570ddb57dfb8b5791500f00f400608cc41eb03e8e637525efee0be59d980"} Oct 30 18:17:09.720385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.720344 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7" event={"ID":"e0b82ac4-2aae-47c4-b032-ab5e463a7920","Type":"ContainerStarted","Data":"bd09a8b32a482f3a579f8d0376bf4b230e4925c86dbd9e2c798f9b64d656fc46"} Oct 30 18:17:09.722010 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.721988 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn" event={"ID":"81cb79ad-89d4-498e-be40-10ee94aa6861","Type":"ContainerStarted","Data":"78151d02332bcb8c4e660a4ae75c0efce5786acf413c8946084e25b53b606488"} Oct 30 18:17:09.768446 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.766335 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7" podStartSLOduration=1.766320884 podStartE2EDuration="1.766320884s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:09.760205781 +0000 UTC m=+929.076626103" watchObservedRunningTime="2026-10-30 18:17:09.766320884 +0000 UTC m=+929.082741217" Oct 30 18:17:09.886956 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.886926 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9" Oct 30 18:17:09.955922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.955886 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/39d2382d-51b2-4f31-aa8e-05205da06225-kube-api-access-tbxpx\" (UniqueName: \"kubernetes.io/projected/39d2382d-51b2-4f31-aa8e-05205da06225-kube-api-access-tbxpx\") pod \"39d2382d-51b2-4f31-aa8e-05205da06225\" (UID: \"39d2382d-51b2-4f31-aa8e-05205da06225\") " Oct 30 18:17:09.956121 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.956046 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/39d2382d-51b2-4f31-aa8e-05205da06225-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d2382d-51b2-4f31-aa8e-05205da06225-maas-api-openshift-ca\") pod \"39d2382d-51b2-4f31-aa8e-05205da06225\" (UID: \"39d2382d-51b2-4f31-aa8e-05205da06225\") " Oct 30 18:17:09.956326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.956221 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/39d2382d-51b2-4f31-aa8e-05205da06225-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d2382d-51b2-4f31-aa8e-05205da06225-maas-api-xks-ca\") pod \"39d2382d-51b2-4f31-aa8e-05205da06225\" (UID: \"39d2382d-51b2-4f31-aa8e-05205da06225\") " Oct 30 18:17:09.958293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.956758 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/39d2382d-51b2-4f31-aa8e-05205da06225-maas-api-openshift-ca" pod "39d2382d-51b2-4f31-aa8e-05205da06225" (UID: "39d2382d-51b2-4f31-aa8e-05205da06225"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:09.960592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.960545 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/39d2382d-51b2-4f31-aa8e-05205da06225-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:09.961242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.961221 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/39d2382d-51b2-4f31-aa8e-05205da06225-maas-api-xks-ca" pod "39d2382d-51b2-4f31-aa8e-05205da06225" (UID: "39d2382d-51b2-4f31-aa8e-05205da06225"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:09.961345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:09.961320 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/39d2382d-51b2-4f31-aa8e-05205da06225-kube-api-access-tbxpx" pod "39d2382d-51b2-4f31-aa8e-05205da06225" (UID: "39d2382d-51b2-4f31-aa8e-05205da06225"). InnerVolumeSpecName "kube-api-access-tbxpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:10.062968 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:10.062826 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/39d2382d-51b2-4f31-aa8e-05205da06225-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:10.062968 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:10.062863 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbxpx\" (UniqueName: \"kubernetes.io/projected/39d2382d-51b2-4f31-aa8e-05205da06225-kube-api-access-tbxpx\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:10.743603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:10.743571 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9" Oct 30 18:17:10.743998 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:10.743570 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9" event={"ID":"39d2382d-51b2-4f31-aa8e-05205da06225","Type":"ContainerDied","Data":"d498468979083feb002bbdfcd67880a3f3a6426745dc6f7bffaa255145678481"} Oct 30 18:17:10.743998 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:10.743727 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d498468979083feb002bbdfcd67880a3f3a6426745dc6f7bffaa255145678481" Oct 30 18:17:10.745359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:10.745333 5232 generic.go:356] "Generic (PLEG): container finished" podID="e0b82ac4-2aae-47c4-b032-ab5e463a7920" containerID="9362570ddb57dfb8b5791500f00f400608cc41eb03e8e637525efee0be59d980" exitCode=0 Oct 30 18:17:10.745478 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:10.745411 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7" event={"ID":"e0b82ac4-2aae-47c4-b032-ab5e463a7920","Type":"ContainerDied","Data":"9362570ddb57dfb8b5791500f00f400608cc41eb03e8e637525efee0be59d980"} Oct 30 18:17:10.747180 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:10.747161 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn" event={"ID":"81cb79ad-89d4-498e-be40-10ee94aa6861","Type":"ContainerStarted","Data":"47511684f56117e29a19dc627dd6f3a0534e97b882873c5883f784e0b5045392"} Oct 30 18:17:10.748892 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:10.748873 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" event={"ID":"c5923f01-3342-409d-8cc6-49e03be3a4ab","Type":"ContainerStarted","Data":"2061a643b81527aa1f3829aba1ef6a731b19d1013c718e825d1f7a482d4c0cbf"} Oct 30 18:17:10.748991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:10.748908 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" event={"ID":"c5923f01-3342-409d-8cc6-49e03be3a4ab","Type":"ContainerStarted","Data":"7ac81f87e7d7fa0d084d4babd0ee537328ee36c8932ad305ab18252dcfe078fe"} Oct 30 18:17:10.846845 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:10.846217 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" podStartSLOduration=1.846195592 podStartE2EDuration="1.846195592s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:10.840360856 +0000 UTC m=+930.156781178" watchObservedRunningTime="2026-10-30 18:17:10.846195592 +0000 UTC m=+930.162615915" Oct 30 18:17:10.879988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:10.879934 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn" podStartSLOduration=2.879919954 podStartE2EDuration="2.879919954s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:10.879006367 +0000 UTC m=+930.195426689" watchObservedRunningTime="2026-10-30 18:17:10.879919954 +0000 UTC m=+930.196340275" Oct 30 18:17:11.393744 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:11.393714 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:11.395078 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:11.395046 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 18:17:11.395214 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:11.395107 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" podUID="c5923f01-3342-409d-8cc6-49e03be3a4ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 18:17:11.759340 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:11.759303 5232 generic.go:356] "Generic (PLEG): container finished" podID="81cb79ad-89d4-498e-be40-10ee94aa6861" containerID="47511684f56117e29a19dc627dd6f3a0534e97b882873c5883f784e0b5045392" exitCode=0 Oct 30 18:17:11.759787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:11.759387 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn" event={"ID":"81cb79ad-89d4-498e-be40-10ee94aa6861","Type":"ContainerDied","Data":"47511684f56117e29a19dc627dd6f3a0534e97b882873c5883f784e0b5045392"} Oct 30 18:17:12.171556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:12.171484 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7" Oct 30 18:17:12.221545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:12.209647 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0b82ac4-2aae-47c4-b032-ab5e463a7920-kube-api-access-82pcv\" (UniqueName: \"kubernetes.io/projected/e0b82ac4-2aae-47c4-b032-ab5e463a7920-kube-api-access-82pcv\") pod \"e0b82ac4-2aae-47c4-b032-ab5e463a7920\" (UID: \"e0b82ac4-2aae-47c4-b032-ab5e463a7920\") " Oct 30 18:17:12.234535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:12.233668 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0b82ac4-2aae-47c4-b032-ab5e463a7920-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b82ac4-2aae-47c4-b032-ab5e463a7920-maas-api-xks-ca\") pod \"e0b82ac4-2aae-47c4-b032-ab5e463a7920\" (UID: \"e0b82ac4-2aae-47c4-b032-ab5e463a7920\") " Oct 30 18:17:12.234535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:12.233849 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0b82ac4-2aae-47c4-b032-ab5e463a7920-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b82ac4-2aae-47c4-b032-ab5e463a7920-maas-api-openshift-ca\") pod \"e0b82ac4-2aae-47c4-b032-ab5e463a7920\" (UID: \"e0b82ac4-2aae-47c4-b032-ab5e463a7920\") " Oct 30 18:17:12.243738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:12.241889 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0b82ac4-2aae-47c4-b032-ab5e463a7920-maas-api-openshift-ca" pod "e0b82ac4-2aae-47c4-b032-ab5e463a7920" (UID: "e0b82ac4-2aae-47c4-b032-ab5e463a7920"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:12.243738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:12.242894 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0b82ac4-2aae-47c4-b032-ab5e463a7920-kube-api-access-82pcv" pod "e0b82ac4-2aae-47c4-b032-ab5e463a7920" (UID: "e0b82ac4-2aae-47c4-b032-ab5e463a7920"). InnerVolumeSpecName "kube-api-access-82pcv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:12.245530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:12.245262 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0b82ac4-2aae-47c4-b032-ab5e463a7920-maas-api-xks-ca" pod "e0b82ac4-2aae-47c4-b032-ab5e463a7920" (UID: "e0b82ac4-2aae-47c4-b032-ab5e463a7920"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:12.336649 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:12.336612 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-82pcv\" (UniqueName: \"kubernetes.io/projected/e0b82ac4-2aae-47c4-b032-ab5e463a7920-kube-api-access-82pcv\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:12.336649 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:12.336643 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0b82ac4-2aae-47c4-b032-ab5e463a7920-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:12.336649 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:12.336652 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0b82ac4-2aae-47c4-b032-ab5e463a7920-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:12.394279 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:12.394192 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 18:17:12.394279 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:12.394247 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" podUID="c5923f01-3342-409d-8cc6-49e03be3a4ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 18:17:12.768207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:12.768178 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7" Oct 30 18:17:12.768619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:12.768210 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7" event={"ID":"e0b82ac4-2aae-47c4-b032-ab5e463a7920","Type":"ContainerDied","Data":"bd09a8b32a482f3a579f8d0376bf4b230e4925c86dbd9e2c798f9b64d656fc46"} Oct 30 18:17:12.768619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:12.768259 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bd09a8b32a482f3a579f8d0376bf4b230e4925c86dbd9e2c798f9b64d656fc46" Oct 30 18:17:12.980284 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:12.980258 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn" Oct 30 18:17:13.046608 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:13.046127 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81cb79ad-89d4-498e-be40-10ee94aa6861-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cb79ad-89d4-498e-be40-10ee94aa6861-maas-api-openshift-ca\") pod \"81cb79ad-89d4-498e-be40-10ee94aa6861\" (UID: \"81cb79ad-89d4-498e-be40-10ee94aa6861\") " Oct 30 18:17:13.046608 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:13.046305 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/81cb79ad-89d4-498e-be40-10ee94aa6861-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cb79ad-89d4-498e-be40-10ee94aa6861-maas-api-xks-ca\") pod \"81cb79ad-89d4-498e-be40-10ee94aa6861\" (UID: \"81cb79ad-89d4-498e-be40-10ee94aa6861\") " Oct 30 18:17:13.046608 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:13.046351 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81cb79ad-89d4-498e-be40-10ee94aa6861-kube-api-access-4dncz\" (UniqueName: \"kubernetes.io/projected/81cb79ad-89d4-498e-be40-10ee94aa6861-kube-api-access-4dncz\") pod \"81cb79ad-89d4-498e-be40-10ee94aa6861\" (UID: \"81cb79ad-89d4-498e-be40-10ee94aa6861\") " Oct 30 18:17:13.046870 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:13.046731 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81cb79ad-89d4-498e-be40-10ee94aa6861-maas-api-openshift-ca" pod "81cb79ad-89d4-498e-be40-10ee94aa6861" (UID: "81cb79ad-89d4-498e-be40-10ee94aa6861"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:13.047098 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:13.047069 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/81cb79ad-89d4-498e-be40-10ee94aa6861-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:13.049410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:13.049384 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/81cb79ad-89d4-498e-be40-10ee94aa6861-maas-api-xks-ca" pod "81cb79ad-89d4-498e-be40-10ee94aa6861" (UID: "81cb79ad-89d4-498e-be40-10ee94aa6861"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:13.049561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:13.049474 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81cb79ad-89d4-498e-be40-10ee94aa6861-kube-api-access-4dncz" pod "81cb79ad-89d4-498e-be40-10ee94aa6861" (UID: "81cb79ad-89d4-498e-be40-10ee94aa6861"). InnerVolumeSpecName "kube-api-access-4dncz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:13.148410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:13.148353 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/81cb79ad-89d4-498e-be40-10ee94aa6861-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:13.148410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:13.148383 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dncz\" (UniqueName: \"kubernetes.io/projected/81cb79ad-89d4-498e-be40-10ee94aa6861-kube-api-access-4dncz\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:13.394121 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:13.394046 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 18:17:13.394269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:13.394119 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" podUID="c5923f01-3342-409d-8cc6-49e03be3a4ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 18:17:13.791121 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:13.791085 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn" event={"ID":"81cb79ad-89d4-498e-be40-10ee94aa6861","Type":"ContainerDied","Data":"78151d02332bcb8c4e660a4ae75c0efce5786acf413c8946084e25b53b606488"} Oct 30 18:17:13.791677 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:13.791148 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="78151d02332bcb8c4e660a4ae75c0efce5786acf413c8946084e25b53b606488" Oct 30 18:17:13.791677 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:13.791213 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn" Oct 30 18:17:14.394199 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:14.394163 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 18:17:14.394402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:14.394220 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" podUID="c5923f01-3342-409d-8cc6-49e03be3a4ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 18:17:15.397134 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:15.397099 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 18:17:15.397804 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:15.397769 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" podUID="c5923f01-3342-409d-8cc6-49e03be3a4ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 18:17:16.393800 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.393769 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" start-of-body= Oct 30 18:17:16.393965 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.393831 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" podUID="c5923f01-3342-409d-8cc6-49e03be3a4ab" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.116:15021/healthz/ready\": dial tcp 10.133.0.116:15021: connect: connection refused" Oct 30 18:17:16.578942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.578909 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7"] Oct 30 18:17:16.579614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.579599 5232 cpu_manager.go:414] "containerMap: removing container" podUID="39d2382d-51b2-4f31-aa8e-05205da06225" containerName="revoke-keys" Oct 30 18:17:16.579674 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.579615 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="39d2382d-51b2-4f31-aa8e-05205da06225" containerName="revoke-keys" Oct 30 18:17:16.579674 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.579628 5232 cpu_manager.go:414] "containerMap: removing container" podUID="81cb79ad-89d4-498e-be40-10ee94aa6861" containerName="revoke-keys" Oct 30 18:17:16.579674 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.579633 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81cb79ad-89d4-498e-be40-10ee94aa6861" containerName="revoke-keys" Oct 30 18:17:16.579674 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.579664 5232 cpu_manager.go:414] "containerMap: removing container" podUID="e0b82ac4-2aae-47c4-b032-ab5e463a7920" containerName="revoke-keys" Oct 30 18:17:16.579674 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.579669 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0b82ac4-2aae-47c4-b032-ab5e463a7920" containerName="revoke-keys" Oct 30 18:17:16.579862 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.579763 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="39d2382d-51b2-4f31-aa8e-05205da06225" containerName="revoke-keys" Oct 30 18:17:16.579862 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.579770 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="81cb79ad-89d4-498e-be40-10ee94aa6861" containerName="revoke-keys" Oct 30 18:17:16.579862 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.579781 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0b82ac4-2aae-47c4-b032-ab5e463a7920" containerName="revoke-keys" Oct 30 18:17:16.583801 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.583785 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7" Oct 30 18:17:16.601146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.601113 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7"] Oct 30 18:17:16.607553 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.607064 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:17:16.607553 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.607191 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:17:16.711213 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.711120 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdef74e1-9dpb7\" (UID: \"82dbd158-2d7e-4c05-9933-43d6e8b85a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7" Oct 30 18:17:16.711213 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.711188 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8kgw\" (UniqueName: \"kubernetes.io/projected/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-kube-api-access-k8kgw\") pod \"maas-api-revoke-sub-fdef74e1-9dpb7\" (UID: \"82dbd158-2d7e-4c05-9933-43d6e8b85a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7" Oct 30 18:17:16.711442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.711279 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdef74e1-9dpb7\" (UID: \"82dbd158-2d7e-4c05-9933-43d6e8b85a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7" Oct 30 18:17:16.813564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.813531 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdef74e1-9dpb7\" (UID: \"82dbd158-2d7e-4c05-9933-43d6e8b85a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7" Oct 30 18:17:16.813766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.813618 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8kgw\" (UniqueName: \"kubernetes.io/projected/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-kube-api-access-k8kgw\") pod \"maas-api-revoke-sub-fdef74e1-9dpb7\" (UID: \"82dbd158-2d7e-4c05-9933-43d6e8b85a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7" Oct 30 18:17:16.813766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.813674 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdef74e1-9dpb7\" (UID: \"82dbd158-2d7e-4c05-9933-43d6e8b85a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7" Oct 30 18:17:16.814418 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.814393 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-fdef74e1-9dpb7\" (UID: \"82dbd158-2d7e-4c05-9933-43d6e8b85a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7" Oct 30 18:17:16.816125 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.816104 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-fdef74e1-9dpb7\" (UID: \"82dbd158-2d7e-4c05-9933-43d6e8b85a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7" Oct 30 18:17:16.832811 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.832782 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8kgw\" (UniqueName: \"kubernetes.io/projected/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-kube-api-access-k8kgw\") pod \"maas-api-revoke-sub-fdef74e1-9dpb7\" (UID: \"82dbd158-2d7e-4c05-9933-43d6e8b85a7d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7" Oct 30 18:17:16.900825 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:16.900787 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7" Oct 30 18:17:17.246540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:17.246492 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7"] Oct 30 18:17:17.248755 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:17:17.248681 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82dbd158_2d7e_4c05_9933_43d6e8b85a7d.slice/crio-7f01db583a3eb35af025960df2c66016c0d87bcc4bae8f26f37b88532455c887 WatchSource:0}: Error finding container 7f01db583a3eb35af025960df2c66016c0d87bcc4bae8f26f37b88532455c887: Status 404 returned error can't find the container with id 7f01db583a3eb35af025960df2c66016c0d87bcc4bae8f26f37b88532455c887 Oct 30 18:17:17.397796 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:17.397771 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:17.398256 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:17.398242 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:17.398977 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:17.398960 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:17:17.830644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:17.830608 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7" event={"ID":"82dbd158-2d7e-4c05-9933-43d6e8b85a7d","Type":"ContainerStarted","Data":"d9604a65a31793dc4e37835ca1b7e6e4ea7cce8c41fc14d999414cff0f2f7cd4"} Oct 30 18:17:17.831062 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:17.830661 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7" event={"ID":"82dbd158-2d7e-4c05-9933-43d6e8b85a7d","Type":"ContainerStarted","Data":"7f01db583a3eb35af025960df2c66016c0d87bcc4bae8f26f37b88532455c887"} Oct 30 18:17:17.869078 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:17.868973 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7" podStartSLOduration=1.8689597249999998 podStartE2EDuration="1.868959725s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:17.867677035 +0000 UTC m=+937.184097360" watchObservedRunningTime="2026-10-30 18:17:17.868959725 +0000 UTC m=+937.185380045" Oct 30 18:17:18.896530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:18.895272 5232 generic.go:356] "Generic (PLEG): container finished" podID="82dbd158-2d7e-4c05-9933-43d6e8b85a7d" containerID="d9604a65a31793dc4e37835ca1b7e6e4ea7cce8c41fc14d999414cff0f2f7cd4" exitCode=0 Oct 30 18:17:18.905472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:18.897045 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7" event={"ID":"82dbd158-2d7e-4c05-9933-43d6e8b85a7d","Type":"ContainerDied","Data":"d9604a65a31793dc4e37835ca1b7e6e4ea7cce8c41fc14d999414cff0f2f7cd4"} Oct 30 18:17:20.080014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.079980 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7" Oct 30 18:17:20.130492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.130457 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-maas-api-openshift-ca\") pod \"82dbd158-2d7e-4c05-9933-43d6e8b85a7d\" (UID: \"82dbd158-2d7e-4c05-9933-43d6e8b85a7d\") " Oct 30 18:17:20.130705 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.130603 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-kube-api-access-k8kgw\" (UniqueName: \"kubernetes.io/projected/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-kube-api-access-k8kgw\") pod \"82dbd158-2d7e-4c05-9933-43d6e8b85a7d\" (UID: \"82dbd158-2d7e-4c05-9933-43d6e8b85a7d\") " Oct 30 18:17:20.130766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.130732 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-maas-api-xks-ca\") pod \"82dbd158-2d7e-4c05-9933-43d6e8b85a7d\" (UID: \"82dbd158-2d7e-4c05-9933-43d6e8b85a7d\") " Oct 30 18:17:20.131234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.131193 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-maas-api-openshift-ca" pod "82dbd158-2d7e-4c05-9933-43d6e8b85a7d" (UID: "82dbd158-2d7e-4c05-9933-43d6e8b85a7d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:20.131724 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.131697 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:20.133952 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.133924 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-maas-api-xks-ca" pod "82dbd158-2d7e-4c05-9933-43d6e8b85a7d" (UID: "82dbd158-2d7e-4c05-9933-43d6e8b85a7d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:20.134052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.133965 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-kube-api-access-k8kgw" pod "82dbd158-2d7e-4c05-9933-43d6e8b85a7d" (UID: "82dbd158-2d7e-4c05-9933-43d6e8b85a7d"). InnerVolumeSpecName "kube-api-access-k8kgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:20.234011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.233939 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:20.234011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.233968 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8kgw\" (UniqueName: \"kubernetes.io/projected/82dbd158-2d7e-4c05-9933-43d6e8b85a7d-kube-api-access-k8kgw\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:20.915290 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.915258 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7" event={"ID":"82dbd158-2d7e-4c05-9933-43d6e8b85a7d","Type":"ContainerDied","Data":"7f01db583a3eb35af025960df2c66016c0d87bcc4bae8f26f37b88532455c887"} Oct 30 18:17:20.915458 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.915309 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7f01db583a3eb35af025960df2c66016c0d87bcc4bae8f26f37b88532455c887" Oct 30 18:17:20.915458 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.915366 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7" Oct 30 18:17:20.923378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.923348 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv"] Oct 30 18:17:20.924122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.924102 5232 cpu_manager.go:414] "containerMap: removing container" podUID="82dbd158-2d7e-4c05-9933-43d6e8b85a7d" containerName="revoke-keys" Oct 30 18:17:20.924244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.924122 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82dbd158-2d7e-4c05-9933-43d6e8b85a7d" containerName="revoke-keys" Oct 30 18:17:20.924304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.924269 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="82dbd158-2d7e-4c05-9933-43d6e8b85a7d" containerName="revoke-keys" Oct 30 18:17:20.928373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.928352 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv" Oct 30 18:17:20.948270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.948236 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd02103a-73fc-4e8d-b77d-128241044320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f190c19-9ndlv\" (UID: \"dd02103a-73fc-4e8d-b77d-128241044320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv" Oct 30 18:17:20.948432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.948382 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rvbsk\" (UniqueName: \"kubernetes.io/projected/dd02103a-73fc-4e8d-b77d-128241044320-kube-api-access-rvbsk\") pod \"maas-api-revoke-sub-2f190c19-9ndlv\" (UID: \"dd02103a-73fc-4e8d-b77d-128241044320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv" Oct 30 18:17:20.948665 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.948643 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd02103a-73fc-4e8d-b77d-128241044320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f190c19-9ndlv\" (UID: \"dd02103a-73fc-4e8d-b77d-128241044320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv" Oct 30 18:17:20.949312 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.949293 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv"] Oct 30 18:17:20.954068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.954046 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:17:20.954187 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:20.954053 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:17:21.050066 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:21.050030 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd02103a-73fc-4e8d-b77d-128241044320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f190c19-9ndlv\" (UID: \"dd02103a-73fc-4e8d-b77d-128241044320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv" Oct 30 18:17:21.050234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:21.050163 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd02103a-73fc-4e8d-b77d-128241044320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f190c19-9ndlv\" (UID: \"dd02103a-73fc-4e8d-b77d-128241044320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv" Oct 30 18:17:21.050234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:21.050214 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rvbsk\" (UniqueName: \"kubernetes.io/projected/dd02103a-73fc-4e8d-b77d-128241044320-kube-api-access-rvbsk\") pod \"maas-api-revoke-sub-2f190c19-9ndlv\" (UID: \"dd02103a-73fc-4e8d-b77d-128241044320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv" Oct 30 18:17:21.050820 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:21.050799 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd02103a-73fc-4e8d-b77d-128241044320-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2f190c19-9ndlv\" (UID: \"dd02103a-73fc-4e8d-b77d-128241044320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv" Oct 30 18:17:21.052667 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:21.052637 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd02103a-73fc-4e8d-b77d-128241044320-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2f190c19-9ndlv\" (UID: \"dd02103a-73fc-4e8d-b77d-128241044320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv" Oct 30 18:17:21.072689 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:21.072628 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rvbsk\" (UniqueName: \"kubernetes.io/projected/dd02103a-73fc-4e8d-b77d-128241044320-kube-api-access-rvbsk\") pod \"maas-api-revoke-sub-2f190c19-9ndlv\" (UID: \"dd02103a-73fc-4e8d-b77d-128241044320\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv" Oct 30 18:17:21.247839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:21.247811 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv" Oct 30 18:17:21.548979 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:21.548953 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv"] Oct 30 18:17:21.549108 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:17:21.549086 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd02103a_73fc_4e8d_b77d_128241044320.slice/crio-d6264bdbd4a99c357b6e45bdd4e4dbbc6eee1482b86ba2eb4e3287d3fd4412f7 WatchSource:0}: Error finding container d6264bdbd4a99c357b6e45bdd4e4dbbc6eee1482b86ba2eb4e3287d3fd4412f7: Status 404 returned error can't find the container with id d6264bdbd4a99c357b6e45bdd4e4dbbc6eee1482b86ba2eb4e3287d3fd4412f7 Oct 30 18:17:21.927369 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:21.927336 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv" event={"ID":"dd02103a-73fc-4e8d-b77d-128241044320","Type":"ContainerStarted","Data":"bd71115cf7337925862094d9f89e095ffbaf5bdf742d233b6bdd9b705ec557c1"} Oct 30 18:17:21.927584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:21.927395 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv" event={"ID":"dd02103a-73fc-4e8d-b77d-128241044320","Type":"ContainerStarted","Data":"d6264bdbd4a99c357b6e45bdd4e4dbbc6eee1482b86ba2eb4e3287d3fd4412f7"} Oct 30 18:17:21.965687 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:21.965634 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv" podStartSLOduration=1.965621848 podStartE2EDuration="1.965621848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:21.965174219 +0000 UTC m=+941.281594540" watchObservedRunningTime="2026-10-30 18:17:21.965621848 +0000 UTC m=+941.282042211" Oct 30 18:17:22.937405 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:22.937367 5232 generic.go:356] "Generic (PLEG): container finished" podID="dd02103a-73fc-4e8d-b77d-128241044320" containerID="bd71115cf7337925862094d9f89e095ffbaf5bdf742d233b6bdd9b705ec557c1" exitCode=0 Oct 30 18:17:22.937813 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:22.937456 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv" event={"ID":"dd02103a-73fc-4e8d-b77d-128241044320","Type":"ContainerDied","Data":"bd71115cf7337925862094d9f89e095ffbaf5bdf742d233b6bdd9b705ec557c1"} Oct 30 18:17:24.221643 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:24.207576 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw"] Oct 30 18:17:24.529561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:24.529530 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw"] Oct 30 18:17:24.552280 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:24.550648 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:24.873946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:24.873859 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:24.874218 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:24.874196 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:24.874424 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:24.874405 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:24.898857 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:24.898822 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b090d280-3716-4098-9330-7fd924785b9d-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:24.899268 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:24.899241 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xr7tv\" (UniqueName: \"kubernetes.io/projected/b090d280-3716-4098-9330-7fd924785b9d-kube-api-access-xr7tv\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:24.899480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:24.899458 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b090d280-3716-4098-9330-7fd924785b9d-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:24.899720 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:24.899701 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:24.899925 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:24.899906 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:24.900116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:24.900098 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b090d280-3716-4098-9330-7fd924785b9d-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.056538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.034792 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.056886 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.056853 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.071581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.071530 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.296826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.279643 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b090d280-3716-4098-9330-7fd924785b9d-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.296826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.279852 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xr7tv\" (UniqueName: \"kubernetes.io/projected/b090d280-3716-4098-9330-7fd924785b9d-kube-api-access-xr7tv\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.296826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.279926 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b090d280-3716-4098-9330-7fd924785b9d-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.296826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.280007 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.296826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.280038 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.296826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.280102 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b090d280-3716-4098-9330-7fd924785b9d-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.296826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.280950 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b090d280-3716-4098-9330-7fd924785b9d-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.296826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.073020 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.296826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.073816 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.296826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.106024 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.343528 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.333909 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.412540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.412097 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.475487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.465583 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:25.475487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.453820 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:17:25.483865 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.479829 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:25.557674 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.515745 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b090d280-3716-4098-9330-7fd924785b9d-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.608378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.605157 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b090d280-3716-4098-9330-7fd924785b9d-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.729537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.728774 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:25.738586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.730124 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:17:25.749291 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.749257 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xr7tv\" (UniqueName: \"kubernetes.io/projected/b090d280-3716-4098-9330-7fd924785b9d-kube-api-access-xr7tv\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:25.825718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.825619 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:25.949534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.922731 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:17:25.949534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.922838 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:17:25.949534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:25.922881 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:17:26.009759 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.009717 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv" Oct 30 18:17:26.059537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.036339 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:17:26.060460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.060426 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:26.218555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.205819 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dd02103a-73fc-4e8d-b77d-128241044320-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd02103a-73fc-4e8d-b77d-128241044320-maas-api-openshift-ca\") pod \"dd02103a-73fc-4e8d-b77d-128241044320\" (UID: \"dd02103a-73fc-4e8d-b77d-128241044320\") " Oct 30 18:17:26.218555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.206065 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dd02103a-73fc-4e8d-b77d-128241044320-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd02103a-73fc-4e8d-b77d-128241044320-maas-api-xks-ca\") pod \"dd02103a-73fc-4e8d-b77d-128241044320\" (UID: \"dd02103a-73fc-4e8d-b77d-128241044320\") " Oct 30 18:17:26.218555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.206180 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dd02103a-73fc-4e8d-b77d-128241044320-kube-api-access-rvbsk\" (UniqueName: \"kubernetes.io/projected/dd02103a-73fc-4e8d-b77d-128241044320-kube-api-access-rvbsk\") pod \"dd02103a-73fc-4e8d-b77d-128241044320\" (UID: \"dd02103a-73fc-4e8d-b77d-128241044320\") " Oct 30 18:17:26.218555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.218296 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dd02103a-73fc-4e8d-b77d-128241044320-maas-api-openshift-ca" pod "dd02103a-73fc-4e8d-b77d-128241044320" (UID: "dd02103a-73fc-4e8d-b77d-128241044320"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:26.265536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.227495 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dd02103a-73fc-4e8d-b77d-128241044320-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:26.406671 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.395745 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:26.478355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.478298 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:17:26.479032 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.479012 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:17:26.497584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.497555 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:26.497885 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.497492 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:26.500373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.498663 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:17:26.500373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.498726 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:17:26.548854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.547584 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dd02103a-73fc-4e8d-b77d-128241044320-maas-api-xks-ca" pod "dd02103a-73fc-4e8d-b77d-128241044320" (UID: "dd02103a-73fc-4e8d-b77d-128241044320"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:26.548854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.548432 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dd02103a-73fc-4e8d-b77d-128241044320-kube-api-access-rvbsk" pod "dd02103a-73fc-4e8d-b77d-128241044320" (UID: "dd02103a-73fc-4e8d-b77d-128241044320"). InnerVolumeSpecName "kube-api-access-rvbsk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:26.610672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.610408 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dd02103a-73fc-4e8d-b77d-128241044320-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:26.610672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.610445 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rvbsk\" (UniqueName: \"kubernetes.io/projected/dd02103a-73fc-4e8d-b77d-128241044320-kube-api-access-rvbsk\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:26.829537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.812852 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:17:26.865537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.856132 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:26.865537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.858435 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 18:17:26.957535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.861424 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:17:26.957535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.946541 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:17:26.957535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:26.946894 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:17:27.337697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:27.337648 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:17:27.379148 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:27.339880 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:17:27.383588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:27.339976 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:17:27.728551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:27.728433 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:17:27.731129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:27.731098 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:17:27.852767 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:27.852687 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:27.867098 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:27.860555 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:27.867098 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:27.861334 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:27.957547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:27.951087 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv" Oct 30 18:17:28.166549 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:17:28.165146 5232 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.218s" Oct 30 18:17:28.166549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.165242 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:17:28.166549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.165293 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:17:28.166549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.165311 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv" event={"ID":"dd02103a-73fc-4e8d-b77d-128241044320","Type":"ContainerDied","Data":"d6264bdbd4a99c357b6e45bdd4e4dbbc6eee1482b86ba2eb4e3287d3fd4412f7"} Oct 30 18:17:28.166549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.165352 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d6264bdbd4a99c357b6e45bdd4e4dbbc6eee1482b86ba2eb4e3287d3fd4412f7" Oct 30 18:17:28.166549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.165436 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:17:28.166549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.165473 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:17:28.166549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.165530 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:17:28.252535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.227262 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.252535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.227678 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:17:28.252535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.228452 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.295557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.273153 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.361954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.361916 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Oct 30 18:17:28.529641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.529597 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": context deadline exceeded" Oct 30 18:17:28.529879 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.529753 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" podUID="1b2b3e72-d4ac-4407-97ae-421c5ad091a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": context deadline exceeded" Oct 30 18:17:28.538550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.532798 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.538550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.533006 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.538550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.533036 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.657535 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:17:28.585126 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:17:28.657535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.595620 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" podUID="693cdf78-86cc-4f53-9c17-3926b662aeb4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.731536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.729629 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:17:28.731536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.731091 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.743541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.733244 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.925119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.925059 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:17:28.969594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.952059 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:28.969594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:28.952375 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:17:29.459808 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:29.459762 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": context deadline exceeded" Oct 30 18:17:29.461211 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:29.461175 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:17:29.514196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:29.471784 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:17:29.528571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:29.471838 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:17:29.528571 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:17:29.475877 5232 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd02103a_73fc_4e8d_b77d_128241044320.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:17:29.528876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:29.479643 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:29.753585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:29.538603 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:17:29.753585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:29.538664 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:29.753585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:29.538691 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:17:29.753585 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:17:29.552400 5232 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd02103a_73fc_4e8d_b77d_128241044320.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:17:29.758559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:29.730464 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:29.877537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:29.761592 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:17:29.943536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:29.933938 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:29.971551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:29.961605 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:29.971551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:29.962909 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 18:17:29.971551 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:17:29.964200 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb090d280_3716_4098_9330_7fd924785b9d.slice/crio-1fef86f30660f7a268750342ef36ffc2cb0740ca4fef1f595e5758de3174d617 WatchSource:0}: Error finding container 1fef86f30660f7a268750342ef36ffc2cb0740ca4fef1f595e5758de3174d617: Status 404 returned error can't find the container with id 1fef86f30660f7a268750342ef36ffc2cb0740ca4fef1f595e5758de3174d617 Oct 30 18:17:29.983596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:29.978703 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:30.029036 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:30.028971 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:17:30.033757 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:30.033703 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:17:30.039038 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:30.038995 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:17:30.065116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:30.065064 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:30.105567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:30.090276 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": context deadline exceeded" Oct 30 18:17:30.537593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:30.513237 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:30.540666 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:30.539246 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:17:30.667653 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:30.664641 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:17:30.734536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:30.733615 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:30.745727 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:30.745683 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:17:30.746328 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:30.746297 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:17:30.765965 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:30.765929 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:17:30.773440 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:30.773404 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:17:30.939593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:30.938620 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:30.990090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:30.987764 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:17:30.990090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:30.988366 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 18:17:30.990090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:30.988845 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:31.687026 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:31.686973 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" event={"ID":"b090d280-3716-4098-9330-7fd924785b9d","Type":"ContainerStarted","Data":"1fef86f30660f7a268750342ef36ffc2cb0740ca4fef1f595e5758de3174d617"} Oct 30 18:17:31.687345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:31.687324 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw"] Oct 30 18:17:31.792676 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:31.792629 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:17:31.944702 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:31.944583 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:17:31.961767 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:31.957257 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:32.012005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.011946 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:32.075052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.066720 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:17:32.080469 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.080208 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:32.083204 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.083164 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:17:32.160356 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.143830 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:32.160356 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.147294 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:17:32.164725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.161997 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6"] Oct 30 18:17:32.164725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.163086 5232 cpu_manager.go:414] "containerMap: removing container" podUID="dd02103a-73fc-4e8d-b77d-128241044320" containerName="revoke-keys" Oct 30 18:17:32.164725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.163107 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dd02103a-73fc-4e8d-b77d-128241044320" containerName="revoke-keys" Oct 30 18:17:32.164725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.163266 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="dd02103a-73fc-4e8d-b77d-128241044320" containerName="revoke-keys" Oct 30 18:17:32.169238 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.169215 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:17:32.169407 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.169396 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6"] Oct 30 18:17:32.169545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.169535 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:17:32.169648 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.169639 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:17:32.169752 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.169743 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:17:32.170756 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.170738 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6" Oct 30 18:17:32.174114 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.173746 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:17:32.176569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.176548 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9"] Oct 30 18:17:32.186109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.186091 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9"] Oct 30 18:17:32.186285 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.186264 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9" Oct 30 18:17:32.186451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.186426 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:17:32.186592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.186582 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:17:32.186725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.186715 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:17:32.186927 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.186854 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:17:32.189051 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.187496 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:17:32.189051 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.187583 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:17:32.202209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.201770 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" event={"ID":"b090d280-3716-4098-9330-7fd924785b9d","Type":"ContainerStarted","Data":"4f27afc9fb40171ec21b0c5657cf3d798240d87e1fcf0621a1b6ce9556682cb5"} Oct 30 18:17:32.212801 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.212775 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97736456-4f31-4954-94f2-f9500b85dc8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd542ec-qmpd6\" (UID: \"97736456-4f31-4954-94f2-f9500b85dc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6" Oct 30 18:17:32.213032 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.213014 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pnhxb\" (UniqueName: \"kubernetes.io/projected/97736456-4f31-4954-94f2-f9500b85dc8c-kube-api-access-pnhxb\") pod \"maas-api-revoke-sub-bfd542ec-qmpd6\" (UID: \"97736456-4f31-4954-94f2-f9500b85dc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6" Oct 30 18:17:32.213228 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.213200 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97736456-4f31-4954-94f2-f9500b85dc8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd542ec-qmpd6\" (UID: \"97736456-4f31-4954-94f2-f9500b85dc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6" Oct 30 18:17:32.314632 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.314595 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pnhxb\" (UniqueName: \"kubernetes.io/projected/97736456-4f31-4954-94f2-f9500b85dc8c-kube-api-access-pnhxb\") pod \"maas-api-revoke-sub-bfd542ec-qmpd6\" (UID: \"97736456-4f31-4954-94f2-f9500b85dc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6" Oct 30 18:17:32.314826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.314651 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa62fecb-fdca-4f29-b713-0bfbb78178f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8ac6bc1-7q9n9\" (UID: \"aa62fecb-fdca-4f29-b713-0bfbb78178f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9" Oct 30 18:17:32.314826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.314766 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97736456-4f31-4954-94f2-f9500b85dc8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd542ec-qmpd6\" (UID: \"97736456-4f31-4954-94f2-f9500b85dc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6" Oct 30 18:17:32.314826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.314804 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twhzk\" (UniqueName: \"kubernetes.io/projected/aa62fecb-fdca-4f29-b713-0bfbb78178f9-kube-api-access-twhzk\") pod \"maas-api-revoke-sub-e8ac6bc1-7q9n9\" (UID: \"aa62fecb-fdca-4f29-b713-0bfbb78178f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9" Oct 30 18:17:32.315333 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.315302 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa62fecb-fdca-4f29-b713-0bfbb78178f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8ac6bc1-7q9n9\" (UID: \"aa62fecb-fdca-4f29-b713-0bfbb78178f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9" Oct 30 18:17:32.316163 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.316135 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97736456-4f31-4954-94f2-f9500b85dc8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd542ec-qmpd6\" (UID: \"97736456-4f31-4954-94f2-f9500b85dc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6" Oct 30 18:17:32.316603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.316584 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97736456-4f31-4954-94f2-f9500b85dc8c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bfd542ec-qmpd6\" (UID: \"97736456-4f31-4954-94f2-f9500b85dc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6" Oct 30 18:17:32.318650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.318632 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97736456-4f31-4954-94f2-f9500b85dc8c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bfd542ec-qmpd6\" (UID: \"97736456-4f31-4954-94f2-f9500b85dc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6" Oct 30 18:17:32.340801 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.340772 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pnhxb\" (UniqueName: \"kubernetes.io/projected/97736456-4f31-4954-94f2-f9500b85dc8c-kube-api-access-pnhxb\") pod \"maas-api-revoke-sub-bfd542ec-qmpd6\" (UID: \"97736456-4f31-4954-94f2-f9500b85dc8c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6" Oct 30 18:17:32.367990 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.367924 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" podStartSLOduration=8.367904648 podStartE2EDuration="8.367904648s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:32.36481491 +0000 UTC m=+951.681235232" watchObservedRunningTime="2026-10-30 18:17:32.367904648 +0000 UTC m=+951.684324971" Oct 30 18:17:32.417330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.417298 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa62fecb-fdca-4f29-b713-0bfbb78178f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8ac6bc1-7q9n9\" (UID: \"aa62fecb-fdca-4f29-b713-0bfbb78178f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9" Oct 30 18:17:32.417503 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.417398 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twhzk\" (UniqueName: \"kubernetes.io/projected/aa62fecb-fdca-4f29-b713-0bfbb78178f9-kube-api-access-twhzk\") pod \"maas-api-revoke-sub-e8ac6bc1-7q9n9\" (UID: \"aa62fecb-fdca-4f29-b713-0bfbb78178f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9" Oct 30 18:17:32.417634 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.417589 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa62fecb-fdca-4f29-b713-0bfbb78178f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8ac6bc1-7q9n9\" (UID: \"aa62fecb-fdca-4f29-b713-0bfbb78178f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9" Oct 30 18:17:32.418024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.418005 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa62fecb-fdca-4f29-b713-0bfbb78178f9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e8ac6bc1-7q9n9\" (UID: \"aa62fecb-fdca-4f29-b713-0bfbb78178f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9" Oct 30 18:17:32.420374 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.420350 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa62fecb-fdca-4f29-b713-0bfbb78178f9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e8ac6bc1-7q9n9\" (UID: \"aa62fecb-fdca-4f29-b713-0bfbb78178f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9" Oct 30 18:17:32.436699 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.436675 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twhzk\" (UniqueName: \"kubernetes.io/projected/aa62fecb-fdca-4f29-b713-0bfbb78178f9-kube-api-access-twhzk\") pod \"maas-api-revoke-sub-e8ac6bc1-7q9n9\" (UID: \"aa62fecb-fdca-4f29-b713-0bfbb78178f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9" Oct 30 18:17:32.501160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.501126 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6" Oct 30 18:17:32.508974 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.508945 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9" Oct 30 18:17:32.882966 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:17:32.882917 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod97736456_4f31_4954_94f2_f9500b85dc8c.slice/crio-66a1f33b311d57d891e69b59ca0ee3ed7d4bf0675c66ea64cb881053f1334db7 WatchSource:0}: Error finding container 66a1f33b311d57d891e69b59ca0ee3ed7d4bf0675c66ea64cb881053f1334db7: Status 404 returned error can't find the container with id 66a1f33b311d57d891e69b59ca0ee3ed7d4bf0675c66ea64cb881053f1334db7 Oct 30 18:17:32.886055 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.886035 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6"] Oct 30 18:17:32.891783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:32.891751 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9"] Oct 30 18:17:33.061921 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:33.061887 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:33.063641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:33.063613 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:17:33.063778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:33.063661 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" podUID="b090d280-3716-4098-9330-7fd924785b9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 18:17:33.236914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:33.236874 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9" event={"ID":"aa62fecb-fdca-4f29-b713-0bfbb78178f9","Type":"ContainerStarted","Data":"d60a72ed6093f63b7b842b55d4bf9642fcff50ca93aed3eeb058dba59b415b79"} Oct 30 18:17:33.238966 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:33.238894 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6" event={"ID":"97736456-4f31-4954-94f2-f9500b85dc8c","Type":"ContainerStarted","Data":"66a1f33b311d57d891e69b59ca0ee3ed7d4bf0675c66ea64cb881053f1334db7"} Oct 30 18:17:34.061589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:34.061551 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:17:34.061969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:34.061605 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" podUID="b090d280-3716-4098-9330-7fd924785b9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 18:17:34.250431 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:34.250393 5232 generic.go:356] "Generic (PLEG): container finished" podID="aa62fecb-fdca-4f29-b713-0bfbb78178f9" containerID="f7831da1c6a0eaa1c0005c8a86deee45a7e77459e3b9edf3dfff91079a9bab3d" exitCode=0 Oct 30 18:17:34.250640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:34.250500 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9" event={"ID":"aa62fecb-fdca-4f29-b713-0bfbb78178f9","Type":"ContainerDied","Data":"f7831da1c6a0eaa1c0005c8a86deee45a7e77459e3b9edf3dfff91079a9bab3d"} Oct 30 18:17:34.252473 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:34.252446 5232 generic.go:356] "Generic (PLEG): container finished" podID="97736456-4f31-4954-94f2-f9500b85dc8c" containerID="99b55c1c67b8a7e76d76e29d3ffa13334b02da569f35c885d013eba7ea960965" exitCode=0 Oct 30 18:17:34.252590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:34.252547 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6" event={"ID":"97736456-4f31-4954-94f2-f9500b85dc8c","Type":"ContainerDied","Data":"99b55c1c67b8a7e76d76e29d3ffa13334b02da569f35c885d013eba7ea960965"} Oct 30 18:17:35.061357 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.061320 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:17:35.061568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.061373 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" podUID="b090d280-3716-4098-9330-7fd924785b9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 18:17:35.696274 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.696243 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9" Oct 30 18:17:35.699800 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.699778 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6" Oct 30 18:17:35.756370 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.756335 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97736456-4f31-4954-94f2-f9500b85dc8c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97736456-4f31-4954-94f2-f9500b85dc8c-maas-api-openshift-ca\") pod \"97736456-4f31-4954-94f2-f9500b85dc8c\" (UID: \"97736456-4f31-4954-94f2-f9500b85dc8c\") " Oct 30 18:17:35.756578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.756453 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/97736456-4f31-4954-94f2-f9500b85dc8c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97736456-4f31-4954-94f2-f9500b85dc8c-maas-api-xks-ca\") pod \"97736456-4f31-4954-94f2-f9500b85dc8c\" (UID: \"97736456-4f31-4954-94f2-f9500b85dc8c\") " Oct 30 18:17:35.756578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.756559 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aa62fecb-fdca-4f29-b713-0bfbb78178f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa62fecb-fdca-4f29-b713-0bfbb78178f9-maas-api-xks-ca\") pod \"aa62fecb-fdca-4f29-b713-0bfbb78178f9\" (UID: \"aa62fecb-fdca-4f29-b713-0bfbb78178f9\") " Oct 30 18:17:35.756743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.756726 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa62fecb-fdca-4f29-b713-0bfbb78178f9-kube-api-access-twhzk\" (UniqueName: \"kubernetes.io/projected/aa62fecb-fdca-4f29-b713-0bfbb78178f9-kube-api-access-twhzk\") pod \"aa62fecb-fdca-4f29-b713-0bfbb78178f9\" (UID: \"aa62fecb-fdca-4f29-b713-0bfbb78178f9\") " Oct 30 18:17:35.756790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.756772 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97736456-4f31-4954-94f2-f9500b85dc8c-maas-api-openshift-ca" pod "97736456-4f31-4954-94f2-f9500b85dc8c" (UID: "97736456-4f31-4954-94f2-f9500b85dc8c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:35.756877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.756858 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa62fecb-fdca-4f29-b713-0bfbb78178f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa62fecb-fdca-4f29-b713-0bfbb78178f9-maas-api-openshift-ca\") pod \"aa62fecb-fdca-4f29-b713-0bfbb78178f9\" (UID: \"aa62fecb-fdca-4f29-b713-0bfbb78178f9\") " Oct 30 18:17:35.757007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.756990 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97736456-4f31-4954-94f2-f9500b85dc8c-kube-api-access-pnhxb\" (UniqueName: \"kubernetes.io/projected/97736456-4f31-4954-94f2-f9500b85dc8c-kube-api-access-pnhxb\") pod \"97736456-4f31-4954-94f2-f9500b85dc8c\" (UID: \"97736456-4f31-4954-94f2-f9500b85dc8c\") " Oct 30 18:17:35.757232 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.757212 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa62fecb-fdca-4f29-b713-0bfbb78178f9-maas-api-openshift-ca" pod "aa62fecb-fdca-4f29-b713-0bfbb78178f9" (UID: "aa62fecb-fdca-4f29-b713-0bfbb78178f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:35.759554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.759011 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/97736456-4f31-4954-94f2-f9500b85dc8c-maas-api-xks-ca" pod "97736456-4f31-4954-94f2-f9500b85dc8c" (UID: "97736456-4f31-4954-94f2-f9500b85dc8c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:35.759554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.759045 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/97736456-4f31-4954-94f2-f9500b85dc8c-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:35.759554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.759093 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aa62fecb-fdca-4f29-b713-0bfbb78178f9-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:35.759554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.759135 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aa62fecb-fdca-4f29-b713-0bfbb78178f9-maas-api-xks-ca" pod "aa62fecb-fdca-4f29-b713-0bfbb78178f9" (UID: "aa62fecb-fdca-4f29-b713-0bfbb78178f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:35.759813 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.759629 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97736456-4f31-4954-94f2-f9500b85dc8c-kube-api-access-pnhxb" pod "97736456-4f31-4954-94f2-f9500b85dc8c" (UID: "97736456-4f31-4954-94f2-f9500b85dc8c"). InnerVolumeSpecName "kube-api-access-pnhxb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:35.778319 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.777734 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa62fecb-fdca-4f29-b713-0bfbb78178f9-kube-api-access-twhzk" pod "aa62fecb-fdca-4f29-b713-0bfbb78178f9" (UID: "aa62fecb-fdca-4f29-b713-0bfbb78178f9"). InnerVolumeSpecName "kube-api-access-twhzk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:35.861146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.861112 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pnhxb\" (UniqueName: \"kubernetes.io/projected/97736456-4f31-4954-94f2-f9500b85dc8c-kube-api-access-pnhxb\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:35.861146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.861145 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/97736456-4f31-4954-94f2-f9500b85dc8c-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:35.861416 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.861156 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aa62fecb-fdca-4f29-b713-0bfbb78178f9-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:35.861416 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.861165 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twhzk\" (UniqueName: \"kubernetes.io/projected/aa62fecb-fdca-4f29-b713-0bfbb78178f9-kube-api-access-twhzk\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:35.875399 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.875359 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb"] Oct 30 18:17:35.876137 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.876115 5232 cpu_manager.go:414] "containerMap: removing container" podUID="aa62fecb-fdca-4f29-b713-0bfbb78178f9" containerName="revoke-keys" Oct 30 18:17:35.876137 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.876138 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa62fecb-fdca-4f29-b713-0bfbb78178f9" containerName="revoke-keys" Oct 30 18:17:35.876305 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.876171 5232 cpu_manager.go:414] "containerMap: removing container" podUID="97736456-4f31-4954-94f2-f9500b85dc8c" containerName="revoke-keys" Oct 30 18:17:35.876305 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.876179 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97736456-4f31-4954-94f2-f9500b85dc8c" containerName="revoke-keys" Oct 30 18:17:35.876410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.876352 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa62fecb-fdca-4f29-b713-0bfbb78178f9" containerName="revoke-keys" Oct 30 18:17:35.876410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.876371 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="97736456-4f31-4954-94f2-f9500b85dc8c" containerName="revoke-keys" Oct 30 18:17:35.880841 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.880818 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb" Oct 30 18:17:35.896761 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.896736 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb"] Oct 30 18:17:35.963248 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.963155 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mm9qq\" (UniqueName: \"kubernetes.io/projected/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-kube-api-access-mm9qq\") pod \"maas-api-revoke-sub-9196a400-x77kb\" (UID: \"1e4bee2a-07c4-4a0b-87ae-16cbd341de21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb" Oct 30 18:17:35.963889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.963550 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9196a400-x77kb\" (UID: \"1e4bee2a-07c4-4a0b-87ae-16cbd341de21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb" Oct 30 18:17:35.963889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:35.963631 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9196a400-x77kb\" (UID: \"1e4bee2a-07c4-4a0b-87ae-16cbd341de21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb" Oct 30 18:17:36.062255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:36.062223 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:17:36.062426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:36.062282 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" podUID="b090d280-3716-4098-9330-7fd924785b9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 18:17:36.065275 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:36.065248 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mm9qq\" (UniqueName: \"kubernetes.io/projected/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-kube-api-access-mm9qq\") pod \"maas-api-revoke-sub-9196a400-x77kb\" (UID: \"1e4bee2a-07c4-4a0b-87ae-16cbd341de21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb" Oct 30 18:17:36.065406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:36.065311 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9196a400-x77kb\" (UID: \"1e4bee2a-07c4-4a0b-87ae-16cbd341de21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb" Oct 30 18:17:36.065406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:36.065351 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9196a400-x77kb\" (UID: \"1e4bee2a-07c4-4a0b-87ae-16cbd341de21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb" Oct 30 18:17:36.066093 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:36.066067 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9196a400-x77kb\" (UID: \"1e4bee2a-07c4-4a0b-87ae-16cbd341de21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb" Oct 30 18:17:36.068105 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:36.068055 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9196a400-x77kb\" (UID: \"1e4bee2a-07c4-4a0b-87ae-16cbd341de21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb" Oct 30 18:17:36.088315 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:36.088273 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mm9qq\" (UniqueName: \"kubernetes.io/projected/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-kube-api-access-mm9qq\") pod \"maas-api-revoke-sub-9196a400-x77kb\" (UID: \"1e4bee2a-07c4-4a0b-87ae-16cbd341de21\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb" Oct 30 18:17:36.200203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:36.200168 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb" Oct 30 18:17:36.270983 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:36.270956 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6" Oct 30 18:17:36.271178 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:36.271009 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6" event={"ID":"97736456-4f31-4954-94f2-f9500b85dc8c","Type":"ContainerDied","Data":"66a1f33b311d57d891e69b59ca0ee3ed7d4bf0675c66ea64cb881053f1334db7"} Oct 30 18:17:36.271178 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:36.271071 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="66a1f33b311d57d891e69b59ca0ee3ed7d4bf0675c66ea64cb881053f1334db7" Oct 30 18:17:36.274150 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:36.273166 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9" event={"ID":"aa62fecb-fdca-4f29-b713-0bfbb78178f9","Type":"ContainerDied","Data":"d60a72ed6093f63b7b842b55d4bf9642fcff50ca93aed3eeb058dba59b415b79"} Oct 30 18:17:36.274150 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:36.273211 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d60a72ed6093f63b7b842b55d4bf9642fcff50ca93aed3eeb058dba59b415b79" Oct 30 18:17:36.274150 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:36.273266 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9" Oct 30 18:17:36.542079 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:17:36.542037 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e4bee2a_07c4_4a0b_87ae_16cbd341de21.slice/crio-424838c5881865bdd2eef1edcf75fda7c1fa5cf0dfafc3efeae54dca61b422c3 WatchSource:0}: Error finding container 424838c5881865bdd2eef1edcf75fda7c1fa5cf0dfafc3efeae54dca61b422c3: Status 404 returned error can't find the container with id 424838c5881865bdd2eef1edcf75fda7c1fa5cf0dfafc3efeae54dca61b422c3 Oct 30 18:17:36.545225 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:36.544766 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb"] Oct 30 18:17:37.061402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:37.061364 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:17:37.061812 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:37.061438 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" podUID="b090d280-3716-4098-9330-7fd924785b9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 18:17:37.333533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:37.325637 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb" event={"ID":"1e4bee2a-07c4-4a0b-87ae-16cbd341de21","Type":"ContainerStarted","Data":"f7aa6e1a61763531b99e5980d1a7c0d66be0ad9030398bb603a5d222ef1eeaf1"} Oct 30 18:17:37.333533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:37.325707 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb" event={"ID":"1e4bee2a-07c4-4a0b-87ae-16cbd341de21","Type":"ContainerStarted","Data":"424838c5881865bdd2eef1edcf75fda7c1fa5cf0dfafc3efeae54dca61b422c3"} Oct 30 18:17:37.368285 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:37.368231 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb" podStartSLOduration=2.368217162 podStartE2EDuration="2.368217162s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:37.365107698 +0000 UTC m=+956.681528004" watchObservedRunningTime="2026-10-30 18:17:37.368217162 +0000 UTC m=+956.684637482" Oct 30 18:17:38.061639 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:38.061604 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:17:38.062194 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:38.061660 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" podUID="b090d280-3716-4098-9330-7fd924785b9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 18:17:38.337282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:38.337194 5232 generic.go:356] "Generic (PLEG): container finished" podID="1e4bee2a-07c4-4a0b-87ae-16cbd341de21" containerID="f7aa6e1a61763531b99e5980d1a7c0d66be0ad9030398bb603a5d222ef1eeaf1" exitCode=0 Oct 30 18:17:38.337441 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:38.337286 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb" event={"ID":"1e4bee2a-07c4-4a0b-87ae-16cbd341de21","Type":"ContainerDied","Data":"f7aa6e1a61763531b99e5980d1a7c0d66be0ad9030398bb603a5d222ef1eeaf1"} Oct 30 18:17:39.064447 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.064415 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" start-of-body= Oct 30 18:17:39.065043 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.065015 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" podUID="b090d280-3716-4098-9330-7fd924785b9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": dial tcp 10.133.0.119:15021: connect: connection refused" Oct 30 18:17:39.535107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.535079 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb" Oct 30 18:17:39.593615 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.593578 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz"] Oct 30 18:17:39.594626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.594582 5232 cpu_manager.go:414] "containerMap: removing container" podUID="1e4bee2a-07c4-4a0b-87ae-16cbd341de21" containerName="revoke-keys" Oct 30 18:17:39.594626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.594627 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1e4bee2a-07c4-4a0b-87ae-16cbd341de21" containerName="revoke-keys" Oct 30 18:17:39.594877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.594860 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="1e4bee2a-07c4-4a0b-87ae-16cbd341de21" containerName="revoke-keys" Oct 30 18:17:39.602122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.600574 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz" Oct 30 18:17:39.630403 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.630369 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz"] Oct 30 18:17:39.680355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.680319 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-maas-api-openshift-ca\") pod \"1e4bee2a-07c4-4a0b-87ae-16cbd341de21\" (UID: \"1e4bee2a-07c4-4a0b-87ae-16cbd341de21\") " Oct 30 18:17:39.680563 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.680425 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-kube-api-access-mm9qq\" (UniqueName: \"kubernetes.io/projected/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-kube-api-access-mm9qq\") pod \"1e4bee2a-07c4-4a0b-87ae-16cbd341de21\" (UID: \"1e4bee2a-07c4-4a0b-87ae-16cbd341de21\") " Oct 30 18:17:39.680563 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.680461 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-maas-api-xks-ca\") pod \"1e4bee2a-07c4-4a0b-87ae-16cbd341de21\" (UID: \"1e4bee2a-07c4-4a0b-87ae-16cbd341de21\") " Oct 30 18:17:39.680795 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.680768 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-maas-api-openshift-ca" pod "1e4bee2a-07c4-4a0b-87ae-16cbd341de21" (UID: "1e4bee2a-07c4-4a0b-87ae-16cbd341de21"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:39.681163 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.681146 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:39.682658 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.682634 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-maas-api-xks-ca" pod "1e4bee2a-07c4-4a0b-87ae-16cbd341de21" (UID: "1e4bee2a-07c4-4a0b-87ae-16cbd341de21"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:39.683168 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.683147 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-kube-api-access-mm9qq" pod "1e4bee2a-07c4-4a0b-87ae-16cbd341de21" (UID: "1e4bee2a-07c4-4a0b-87ae-16cbd341de21"). InnerVolumeSpecName "kube-api-access-mm9qq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:39.782662 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.782628 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804c225a-2123-430a-a36d-cf349bf258e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65adf21e-4hvjz\" (UID: \"804c225a-2123-430a-a36d-cf349bf258e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz" Oct 30 18:17:39.782825 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.782691 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5hdn\" (UniqueName: \"kubernetes.io/projected/804c225a-2123-430a-a36d-cf349bf258e7-kube-api-access-m5hdn\") pod \"maas-api-revoke-sub-65adf21e-4hvjz\" (UID: \"804c225a-2123-430a-a36d-cf349bf258e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz" Oct 30 18:17:39.782868 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.782825 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804c225a-2123-430a-a36d-cf349bf258e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65adf21e-4hvjz\" (UID: \"804c225a-2123-430a-a36d-cf349bf258e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz" Oct 30 18:17:39.783014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.782998 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mm9qq\" (UniqueName: \"kubernetes.io/projected/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-kube-api-access-mm9qq\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:39.783051 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.783019 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1e4bee2a-07c4-4a0b-87ae-16cbd341de21-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:39.884188 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.884097 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804c225a-2123-430a-a36d-cf349bf258e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65adf21e-4hvjz\" (UID: \"804c225a-2123-430a-a36d-cf349bf258e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz" Oct 30 18:17:39.884362 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.884188 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5hdn\" (UniqueName: \"kubernetes.io/projected/804c225a-2123-430a-a36d-cf349bf258e7-kube-api-access-m5hdn\") pod \"maas-api-revoke-sub-65adf21e-4hvjz\" (UID: \"804c225a-2123-430a-a36d-cf349bf258e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz" Oct 30 18:17:39.884438 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.884415 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804c225a-2123-430a-a36d-cf349bf258e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65adf21e-4hvjz\" (UID: \"804c225a-2123-430a-a36d-cf349bf258e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz" Oct 30 18:17:39.885354 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.885324 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804c225a-2123-430a-a36d-cf349bf258e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65adf21e-4hvjz\" (UID: \"804c225a-2123-430a-a36d-cf349bf258e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz" Oct 30 18:17:39.886821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.886796 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804c225a-2123-430a-a36d-cf349bf258e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65adf21e-4hvjz\" (UID: \"804c225a-2123-430a-a36d-cf349bf258e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz" Oct 30 18:17:39.903373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.903355 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5hdn\" (UniqueName: \"kubernetes.io/projected/804c225a-2123-430a-a36d-cf349bf258e7-kube-api-access-m5hdn\") pod \"maas-api-revoke-sub-65adf21e-4hvjz\" (UID: \"804c225a-2123-430a-a36d-cf349bf258e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz" Oct 30 18:17:39.927829 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:39.927802 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz" Oct 30 18:17:40.070525 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:40.070470 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:40.071123 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:40.071098 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:40.072542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:40.072485 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:17:40.574522 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:17:40.562044 5232 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod804c225a_2123_430a_a36d_cf349bf258e7.slice/crio-170205e49e9743eb331b76c47fe8c38679ad2c807f63ae4f4a667a6263a7e86e\": RecentStats: unable to find data in memory cache]" Oct 30 18:17:41.372534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:41.356449 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb" Oct 30 18:17:41.372534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:41.369438 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz"] Oct 30 18:17:41.387235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:41.369492 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb" event={"ID":"1e4bee2a-07c4-4a0b-87ae-16cbd341de21","Type":"ContainerDied","Data":"424838c5881865bdd2eef1edcf75fda7c1fa5cf0dfafc3efeae54dca61b422c3"} Oct 30 18:17:41.387235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:41.382616 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="424838c5881865bdd2eef1edcf75fda7c1fa5cf0dfafc3efeae54dca61b422c3" Oct 30 18:17:41.387235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:41.382648 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz" event={"ID":"804c225a-2123-430a-a36d-cf349bf258e7","Type":"ContainerStarted","Data":"170205e49e9743eb331b76c47fe8c38679ad2c807f63ae4f4a667a6263a7e86e"} Oct 30 18:17:41.530814 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:41.530751 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:41.602931 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:41.602836 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:41.648912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:41.646884 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:41.813544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:41.810810 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:17:42.336539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:42.335660 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:17:42.358535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:42.354084 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:42.358535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:42.354673 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:42.358798 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:42.358671 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:17:42.385542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:42.359113 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:17:42.533669 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:42.533617 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:42.551344 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:42.547215 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:17:42.551678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:42.551642 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:42.599602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:42.576868 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz" event={"ID":"804c225a-2123-430a-a36d-cf349bf258e7","Type":"ContainerStarted","Data":"eb16d995657784c42efcf9ed7bb2b0b7ba4d4e315302e2685b6d60a95d175948"} Oct 30 18:17:42.603786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:42.603748 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:42.604703 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:42.604680 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:17:42.671944 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:42.668254 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:17:42.674156 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:42.674114 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:17:42.688674 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:42.688641 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb"] Oct 30 18:17:42.760143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:42.760094 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:42.829143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:42.829098 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:43.031535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.011729 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:17:43.031535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.011833 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7ab538b5-wbkkb"] Oct 30 18:17:43.413922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.351180 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:17:43.413922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.352005 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:17:43.413922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.358052 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:43.413922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.359237 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:17:43.413922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.359617 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:17:43.413922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.359673 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:17:43.451686 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.433614 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:17:43.451686 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.434208 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:17:43.451686 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.439088 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:17:43.451686 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.439161 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:17:43.495560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.495465 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz" podStartSLOduration=4.4954420840000004 podStartE2EDuration="4.495442084s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:42.676233871 +0000 UTC m=+961.992654192" watchObservedRunningTime="2026-10-30 18:17:43.495442084 +0000 UTC m=+962.811862406" Oct 30 18:17:43.537195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.537152 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:17:43.670636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.625948 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:43.670636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.668873 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:17:43.815547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.812271 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:43.850539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.842597 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:17:43.862577 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.850811 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:43.936594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.870267 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:17:43.963742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:43.963707 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:17:44.074322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.072295 5232 generic.go:356] "Generic (PLEG): container finished" podID="804c225a-2123-430a-a36d-cf349bf258e7" containerID="eb16d995657784c42efcf9ed7bb2b0b7ba4d4e315302e2685b6d60a95d175948" exitCode=0 Oct 30 18:17:44.074322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.072453 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz" event={"ID":"804c225a-2123-430a-a36d-cf349bf258e7","Type":"ContainerDied","Data":"eb16d995657784c42efcf9ed7bb2b0b7ba4d4e315302e2685b6d60a95d175948"} Oct 30 18:17:44.364844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.355284 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:44.364844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.355707 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:44.364844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.358988 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:44.364844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.364573 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:44.442140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.441611 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:44.444952 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.444884 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:44.554535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.551614 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:17:44.633056 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.632960 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:44.679159 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.679114 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:17:44.751196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.751161 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2"] Oct 30 18:17:44.825578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.825537 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:17:44.858989 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.858946 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:44.945595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.876177 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:44.949536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.949416 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xnl6f\" (UniqueName: \"kubernetes.io/projected/4dd65e37-2961-45d9-ab25-7db48029ebbe-kube-api-access-xnl6f\") pod \"maas-api-revoke-sub-3984aee6-kz7q2\" (UID: \"4dd65e37-2961-45d9-ab25-7db48029ebbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2" Oct 30 18:17:44.949536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.949478 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd65e37-2961-45d9-ab25-7db48029ebbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3984aee6-kz7q2\" (UID: \"4dd65e37-2961-45d9-ab25-7db48029ebbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2" Oct 30 18:17:44.973551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.949778 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd65e37-2961-45d9-ab25-7db48029ebbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3984aee6-kz7q2\" (UID: \"4dd65e37-2961-45d9-ab25-7db48029ebbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2" Oct 30 18:17:44.973551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:44.960011 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2" Oct 30 18:17:45.159538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.102054 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd65e37-2961-45d9-ab25-7db48029ebbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3984aee6-kz7q2\" (UID: \"4dd65e37-2961-45d9-ab25-7db48029ebbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2" Oct 30 18:17:45.159538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.102307 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xnl6f\" (UniqueName: \"kubernetes.io/projected/4dd65e37-2961-45d9-ab25-7db48029ebbe-kube-api-access-xnl6f\") pod \"maas-api-revoke-sub-3984aee6-kz7q2\" (UID: \"4dd65e37-2961-45d9-ab25-7db48029ebbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2" Oct 30 18:17:45.159538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.102359 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd65e37-2961-45d9-ab25-7db48029ebbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3984aee6-kz7q2\" (UID: \"4dd65e37-2961-45d9-ab25-7db48029ebbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2" Oct 30 18:17:45.159538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.103725 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd65e37-2961-45d9-ab25-7db48029ebbe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3984aee6-kz7q2\" (UID: \"4dd65e37-2961-45d9-ab25-7db48029ebbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2" Oct 30 18:17:45.417536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.266123 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17f67c08-cc43-45c3-995f-ed8c7e4c7708" path="/var/lib/kubelet/pods/17f67c08-cc43-45c3-995f-ed8c7e4c7708/volumes" Oct 30 18:17:45.449575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.325186 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd65e37-2961-45d9-ab25-7db48029ebbe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3984aee6-kz7q2\" (UID: \"4dd65e37-2961-45d9-ab25-7db48029ebbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2" Oct 30 18:17:45.449575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.362892 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:45.449575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.362935 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:17:45.449575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.389195 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:17:45.456637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.456591 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2"] Oct 30 18:17:45.458353 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.456612 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:17:45.458946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.456664 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:45.557724 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.557670 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:45.594884 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.594850 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xnl6f\" (UniqueName: \"kubernetes.io/projected/4dd65e37-2961-45d9-ab25-7db48029ebbe-kube-api-access-xnl6f\") pod \"maas-api-revoke-sub-3984aee6-kz7q2\" (UID: \"4dd65e37-2961-45d9-ab25-7db48029ebbe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2" Oct 30 18:17:45.659919 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.659868 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:45.703613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.684842 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:17:45.870680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.870638 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2" Oct 30 18:17:45.886059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.885998 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:45.886808 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.886763 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:17:45.998606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:45.897233 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:46.221636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.221442 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:17:46.671305 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.671218 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:17:46.673795 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.673765 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:17:46.674488 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.674459 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:17:46.711128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.711095 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:17:46.722542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.712616 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:17:46.724939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.712617 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:17:46.770296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.713405 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 18:17:46.770296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.713447 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:17:46.899685 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.897221 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:17:46.960591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.938056 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:17:46.960591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.938111 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:17:46.960591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.938140 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:17:46.960591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.938170 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:17:46.960591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.938198 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:17:46.960591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.938231 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:17:46.981868 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.970536 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:17:46.981868 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.974612 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:17:46.981868 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:46.979567 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:17:47.143799 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:47.141659 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz" Oct 30 18:17:47.241607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:47.241565 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/804c225a-2123-430a-a36d-cf349bf258e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804c225a-2123-430a-a36d-cf349bf258e7-maas-api-openshift-ca\") pod \"804c225a-2123-430a-a36d-cf349bf258e7\" (UID: \"804c225a-2123-430a-a36d-cf349bf258e7\") " Oct 30 18:17:47.241791 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:47.241685 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/804c225a-2123-430a-a36d-cf349bf258e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804c225a-2123-430a-a36d-cf349bf258e7-maas-api-xks-ca\") pod \"804c225a-2123-430a-a36d-cf349bf258e7\" (UID: \"804c225a-2123-430a-a36d-cf349bf258e7\") " Oct 30 18:17:47.241791 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:47.241719 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/804c225a-2123-430a-a36d-cf349bf258e7-kube-api-access-m5hdn\" (UniqueName: \"kubernetes.io/projected/804c225a-2123-430a-a36d-cf349bf258e7-kube-api-access-m5hdn\") pod \"804c225a-2123-430a-a36d-cf349bf258e7\" (UID: \"804c225a-2123-430a-a36d-cf349bf258e7\") " Oct 30 18:17:47.241977 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:47.241950 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/804c225a-2123-430a-a36d-cf349bf258e7-maas-api-openshift-ca" pod "804c225a-2123-430a-a36d-cf349bf258e7" (UID: "804c225a-2123-430a-a36d-cf349bf258e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:47.242640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:47.242618 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/804c225a-2123-430a-a36d-cf349bf258e7-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:47.244059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:47.244028 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/804c225a-2123-430a-a36d-cf349bf258e7-kube-api-access-m5hdn" pod "804c225a-2123-430a-a36d-cf349bf258e7" (UID: "804c225a-2123-430a-a36d-cf349bf258e7"). InnerVolumeSpecName "kube-api-access-m5hdn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:47.244347 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:47.244322 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/804c225a-2123-430a-a36d-cf349bf258e7-maas-api-xks-ca" pod "804c225a-2123-430a-a36d-cf349bf258e7" (UID: "804c225a-2123-430a-a36d-cf349bf258e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:47.336878 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:47.336850 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2"] Oct 30 18:17:47.337431 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:17:47.337408 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd65e37_2961_45d9_ab25_7db48029ebbe.slice/crio-95fc9e50d5dd08dd4629a3a9e1972866b6ebaed3d9d814ae1bc8b88cab8407b7 WatchSource:0}: Error finding container 95fc9e50d5dd08dd4629a3a9e1972866b6ebaed3d9d814ae1bc8b88cab8407b7: Status 404 returned error can't find the container with id 95fc9e50d5dd08dd4629a3a9e1972866b6ebaed3d9d814ae1bc8b88cab8407b7 Oct 30 18:17:47.348569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:47.345251 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/804c225a-2123-430a-a36d-cf349bf258e7-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:47.348569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:47.345282 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5hdn\" (UniqueName: \"kubernetes.io/projected/804c225a-2123-430a-a36d-cf349bf258e7-kube-api-access-m5hdn\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:47.762535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:47.762478 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz" Oct 30 18:17:47.762976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:47.762529 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz" event={"ID":"804c225a-2123-430a-a36d-cf349bf258e7","Type":"ContainerDied","Data":"170205e49e9743eb331b76c47fe8c38679ad2c807f63ae4f4a667a6263a7e86e"} Oct 30 18:17:47.762976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:47.762576 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="170205e49e9743eb331b76c47fe8c38679ad2c807f63ae4f4a667a6263a7e86e" Oct 30 18:17:47.764578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:47.764544 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2" event={"ID":"4dd65e37-2961-45d9-ab25-7db48029ebbe","Type":"ContainerStarted","Data":"efc2a4ba046c5b48ebb669e05b540084ea4587784f1242ebbde70b22ae74dcf4"} Oct 30 18:17:47.764578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:47.764580 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2" event={"ID":"4dd65e37-2961-45d9-ab25-7db48029ebbe","Type":"ContainerStarted","Data":"95fc9e50d5dd08dd4629a3a9e1972866b6ebaed3d9d814ae1bc8b88cab8407b7"} Oct 30 18:17:47.810475 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:47.810373 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2" podStartSLOduration=4.810357246 podStartE2EDuration="4.810357246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:47.807071314 +0000 UTC m=+967.123491634" watchObservedRunningTime="2026-10-30 18:17:47.810357246 +0000 UTC m=+967.126777566" Oct 30 18:17:48.783832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:48.783794 5232 generic.go:356] "Generic (PLEG): container finished" podID="4dd65e37-2961-45d9-ab25-7db48029ebbe" containerID="efc2a4ba046c5b48ebb669e05b540084ea4587784f1242ebbde70b22ae74dcf4" exitCode=0 Oct 30 18:17:48.784207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:48.783851 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2" event={"ID":"4dd65e37-2961-45d9-ab25-7db48029ebbe","Type":"ContainerDied","Data":"efc2a4ba046c5b48ebb669e05b540084ea4587784f1242ebbde70b22ae74dcf4"} Oct 30 18:17:49.234402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.234318 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2"] Oct 30 18:17:49.235172 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.235155 5232 cpu_manager.go:414] "containerMap: removing container" podUID="804c225a-2123-430a-a36d-cf349bf258e7" containerName="revoke-keys" Oct 30 18:17:49.235238 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.235175 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="804c225a-2123-430a-a36d-cf349bf258e7" containerName="revoke-keys" Oct 30 18:17:49.235311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.235298 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="804c225a-2123-430a-a36d-cf349bf258e7" containerName="revoke-keys" Oct 30 18:17:49.239101 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.239082 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2" Oct 30 18:17:49.256657 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.256630 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2"] Oct 30 18:17:49.285231 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.285204 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e407db5-83a8-4496-843e-0974196bd4d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e70f68c8-k5kg2\" (UID: \"6e407db5-83a8-4496-843e-0974196bd4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2" Oct 30 18:17:49.285407 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.285392 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e407db5-83a8-4496-843e-0974196bd4d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e70f68c8-k5kg2\" (UID: \"6e407db5-83a8-4496-843e-0974196bd4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2" Oct 30 18:17:49.285480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.285439 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lx92x\" (UniqueName: \"kubernetes.io/projected/6e407db5-83a8-4496-843e-0974196bd4d7-kube-api-access-lx92x\") pod \"maas-api-revoke-sub-e70f68c8-k5kg2\" (UID: \"6e407db5-83a8-4496-843e-0974196bd4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2" Oct 30 18:17:49.386638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.386608 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e407db5-83a8-4496-843e-0974196bd4d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e70f68c8-k5kg2\" (UID: \"6e407db5-83a8-4496-843e-0974196bd4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2" Oct 30 18:17:49.386773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.386661 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lx92x\" (UniqueName: \"kubernetes.io/projected/6e407db5-83a8-4496-843e-0974196bd4d7-kube-api-access-lx92x\") pod \"maas-api-revoke-sub-e70f68c8-k5kg2\" (UID: \"6e407db5-83a8-4496-843e-0974196bd4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2" Oct 30 18:17:49.386855 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.386791 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e407db5-83a8-4496-843e-0974196bd4d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e70f68c8-k5kg2\" (UID: \"6e407db5-83a8-4496-843e-0974196bd4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2" Oct 30 18:17:49.387212 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.387190 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e407db5-83a8-4496-843e-0974196bd4d7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e70f68c8-k5kg2\" (UID: \"6e407db5-83a8-4496-843e-0974196bd4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2" Oct 30 18:17:49.389255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.389232 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e407db5-83a8-4496-843e-0974196bd4d7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e70f68c8-k5kg2\" (UID: \"6e407db5-83a8-4496-843e-0974196bd4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2" Oct 30 18:17:49.406085 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.406053 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lx92x\" (UniqueName: \"kubernetes.io/projected/6e407db5-83a8-4496-843e-0974196bd4d7-kube-api-access-lx92x\") pod \"maas-api-revoke-sub-e70f68c8-k5kg2\" (UID: \"6e407db5-83a8-4496-843e-0974196bd4d7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2" Oct 30 18:17:49.559240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.559212 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2" Oct 30 18:17:49.886304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.886269 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2"] Oct 30 18:17:49.943259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.943237 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2" Oct 30 18:17:49.996369 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.996337 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4dd65e37-2961-45d9-ab25-7db48029ebbe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd65e37-2961-45d9-ab25-7db48029ebbe-maas-api-openshift-ca\") pod \"4dd65e37-2961-45d9-ab25-7db48029ebbe\" (UID: \"4dd65e37-2961-45d9-ab25-7db48029ebbe\") " Oct 30 18:17:49.996593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.996437 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4dd65e37-2961-45d9-ab25-7db48029ebbe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd65e37-2961-45d9-ab25-7db48029ebbe-maas-api-xks-ca\") pod \"4dd65e37-2961-45d9-ab25-7db48029ebbe\" (UID: \"4dd65e37-2961-45d9-ab25-7db48029ebbe\") " Oct 30 18:17:49.996593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.996560 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4dd65e37-2961-45d9-ab25-7db48029ebbe-kube-api-access-xnl6f\" (UniqueName: \"kubernetes.io/projected/4dd65e37-2961-45d9-ab25-7db48029ebbe-kube-api-access-xnl6f\") pod \"4dd65e37-2961-45d9-ab25-7db48029ebbe\" (UID: \"4dd65e37-2961-45d9-ab25-7db48029ebbe\") " Oct 30 18:17:49.996831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.996802 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd65e37-2961-45d9-ab25-7db48029ebbe-maas-api-openshift-ca" pod "4dd65e37-2961-45d9-ab25-7db48029ebbe" (UID: "4dd65e37-2961-45d9-ab25-7db48029ebbe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:49.997453 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:49.997427 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4dd65e37-2961-45d9-ab25-7db48029ebbe-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:50.000101 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.000075 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4dd65e37-2961-45d9-ab25-7db48029ebbe-maas-api-xks-ca" pod "4dd65e37-2961-45d9-ab25-7db48029ebbe" (UID: "4dd65e37-2961-45d9-ab25-7db48029ebbe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:50.000376 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.000351 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd65e37-2961-45d9-ab25-7db48029ebbe-kube-api-access-xnl6f" pod "4dd65e37-2961-45d9-ab25-7db48029ebbe" (UID: "4dd65e37-2961-45d9-ab25-7db48029ebbe"). InnerVolumeSpecName "kube-api-access-xnl6f". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:50.099554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.099488 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4dd65e37-2961-45d9-ab25-7db48029ebbe-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:50.099554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.099545 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xnl6f\" (UniqueName: \"kubernetes.io/projected/4dd65e37-2961-45d9-ab25-7db48029ebbe-kube-api-access-xnl6f\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:50.605045 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.605009 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w"] Oct 30 18:17:50.606125 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.606105 5232 cpu_manager.go:414] "containerMap: removing container" podUID="4dd65e37-2961-45d9-ab25-7db48029ebbe" containerName="revoke-keys" Oct 30 18:17:50.606193 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.606128 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4dd65e37-2961-45d9-ab25-7db48029ebbe" containerName="revoke-keys" Oct 30 18:17:50.606334 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.606321 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="4dd65e37-2961-45d9-ab25-7db48029ebbe" containerName="revoke-keys" Oct 30 18:17:50.611241 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.611165 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.639745 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.638695 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-535b743b-gw-openshift-default-dockercfg-wqkzc\"" Oct 30 18:17:50.648801 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.648765 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w"] Oct 30 18:17:50.709387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.709317 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-token\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.709387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.709360 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-data\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.709920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.709638 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6pjkr\" (UniqueName: \"kubernetes.io/projected/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-kube-api-access-6pjkr\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.709920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.709696 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-credential-socket\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.709920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.709821 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-podinfo\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.709920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.709849 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-workload-socket\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.709920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.709872 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-workload-certs\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.710217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.710029 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istiod-ca-cert\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.710217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.710059 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-envoy\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.817982 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.817941 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-token\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.818405 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.818379 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-data\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.818816 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.818789 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6pjkr\" (UniqueName: \"kubernetes.io/projected/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-kube-api-access-6pjkr\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.843634 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.836774 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-data\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.844135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.844112 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-credential-socket\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.879067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.879032 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-podinfo\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.879300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.879282 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-workload-socket\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.879399 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.879386 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-workload-certs\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.879671 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.879654 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istiod-ca-cert\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.879805 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.879791 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-envoy\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.886927 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.886892 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-envoy\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.887468 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.844665 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-credential-socket\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.896771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.895882 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-workload-certs\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.896771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.896135 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-workload-socket\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.896771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.896685 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istiod-ca-cert\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.949338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.933370 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-token\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:50.949338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:50.934847 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-podinfo\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:51.030534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:51.020525 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2" Oct 30 18:17:51.064558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:51.054074 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2" event={"ID":"4dd65e37-2961-45d9-ab25-7db48029ebbe","Type":"ContainerDied","Data":"95fc9e50d5dd08dd4629a3a9e1972866b6ebaed3d9d814ae1bc8b88cab8407b7"} Oct 30 18:17:51.064558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:51.054146 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="95fc9e50d5dd08dd4629a3a9e1972866b6ebaed3d9d814ae1bc8b88cab8407b7" Oct 30 18:17:51.068065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:51.068000 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6pjkr\" (UniqueName: \"kubernetes.io/projected/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-kube-api-access-6pjkr\") pod \"e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:51.078842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:51.078811 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2" event={"ID":"6e407db5-83a8-4496-843e-0974196bd4d7","Type":"ContainerStarted","Data":"bbfb4e4052046f8e756ba31c4fd5dbadb05550a0589c14aa472536a6fae80a00"} Oct 30 18:17:51.079088 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:51.079073 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2" event={"ID":"6e407db5-83a8-4496-843e-0974196bd4d7","Type":"ContainerStarted","Data":"ca18702277c154f39fee51f0772b27d09d2001c4e847197da9b951e8270bcf2e"} Oct 30 18:17:51.235220 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:51.235188 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:51.571769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:51.571743 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w"] Oct 30 18:17:51.571769 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:17:51.571757 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fddc7f1_0ab6_45e0_9a80_6cfbc05a4984.slice/crio-725d430383e74c0649a31aaada638d950b90238bf6c83de19fefd8777e4fd0e7 WatchSource:0}: Error finding container 725d430383e74c0649a31aaada638d950b90238bf6c83de19fefd8777e4fd0e7: Status 404 returned error can't find the container with id 725d430383e74c0649a31aaada638d950b90238bf6c83de19fefd8777e4fd0e7 Oct 30 18:17:51.574452 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:51.574420 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:17:51.574542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:51.574490 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:17:51.574542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:51.574534 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:17:52.089461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:52.089423 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" event={"ID":"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984","Type":"ContainerStarted","Data":"bc9065ec30dd487b69207a9afcb3e5f34eabe7897fc972d52595d651e7a70e7d"} Oct 30 18:17:52.089928 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:52.089483 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" event={"ID":"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984","Type":"ContainerStarted","Data":"725d430383e74c0649a31aaada638d950b90238bf6c83de19fefd8777e4fd0e7"} Oct 30 18:17:52.091567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:52.091524 5232 generic.go:356] "Generic (PLEG): container finished" podID="6e407db5-83a8-4496-843e-0974196bd4d7" containerID="bbfb4e4052046f8e756ba31c4fd5dbadb05550a0589c14aa472536a6fae80a00" exitCode=0 Oct 30 18:17:52.091705 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:52.091602 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2" event={"ID":"6e407db5-83a8-4496-843e-0974196bd4d7","Type":"ContainerDied","Data":"bbfb4e4052046f8e756ba31c4fd5dbadb05550a0589c14aa472536a6fae80a00"} Oct 30 18:17:52.145226 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:52.145163 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" podStartSLOduration=2.145143889 podStartE2EDuration="2.145143889s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:17:52.142297539 +0000 UTC m=+971.458717861" watchObservedRunningTime="2026-10-30 18:17:52.145143889 +0000 UTC m=+971.461564211" Oct 30 18:17:52.235388 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:52.235353 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:52.241861 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:52.241836 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:52.329423 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:52.329379 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2" Oct 30 18:17:52.448343 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:52.448255 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e407db5-83a8-4496-843e-0974196bd4d7-kube-api-access-lx92x\" (UniqueName: \"kubernetes.io/projected/6e407db5-83a8-4496-843e-0974196bd4d7-kube-api-access-lx92x\") pod \"6e407db5-83a8-4496-843e-0974196bd4d7\" (UID: \"6e407db5-83a8-4496-843e-0974196bd4d7\") " Oct 30 18:17:52.448572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:52.448494 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6e407db5-83a8-4496-843e-0974196bd4d7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e407db5-83a8-4496-843e-0974196bd4d7-maas-api-xks-ca\") pod \"6e407db5-83a8-4496-843e-0974196bd4d7\" (UID: \"6e407db5-83a8-4496-843e-0974196bd4d7\") " Oct 30 18:17:52.448684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:52.448662 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6e407db5-83a8-4496-843e-0974196bd4d7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e407db5-83a8-4496-843e-0974196bd4d7-maas-api-openshift-ca\") pod \"6e407db5-83a8-4496-843e-0974196bd4d7\" (UID: \"6e407db5-83a8-4496-843e-0974196bd4d7\") " Oct 30 18:17:52.449827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:52.449795 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6e407db5-83a8-4496-843e-0974196bd4d7-maas-api-openshift-ca" pod "6e407db5-83a8-4496-843e-0974196bd4d7" (UID: "6e407db5-83a8-4496-843e-0974196bd4d7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:17:52.451400 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:52.451369 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e407db5-83a8-4496-843e-0974196bd4d7-kube-api-access-lx92x" pod "6e407db5-83a8-4496-843e-0974196bd4d7" (UID: "6e407db5-83a8-4496-843e-0974196bd4d7"). InnerVolumeSpecName "kube-api-access-lx92x". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:17:52.452703 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:52.452678 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6e407db5-83a8-4496-843e-0974196bd4d7-maas-api-xks-ca" pod "6e407db5-83a8-4496-843e-0974196bd4d7" (UID: "6e407db5-83a8-4496-843e-0974196bd4d7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:17:52.550492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:52.550457 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6e407db5-83a8-4496-843e-0974196bd4d7-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:52.550492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:52.550481 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6e407db5-83a8-4496-843e-0974196bd4d7-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:52.550492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:52.550493 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lx92x\" (UniqueName: \"kubernetes.io/projected/6e407db5-83a8-4496-843e-0974196bd4d7-kube-api-access-lx92x\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:17:53.102910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:53.102829 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2" event={"ID":"6e407db5-83a8-4496-843e-0974196bd4d7","Type":"ContainerDied","Data":"ca18702277c154f39fee51f0772b27d09d2001c4e847197da9b951e8270bcf2e"} Oct 30 18:17:53.102910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:53.102848 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2" Oct 30 18:17:53.102910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:53.102878 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca18702277c154f39fee51f0772b27d09d2001c4e847197da9b951e8270bcf2e" Oct 30 18:17:53.103425 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:53.103242 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:53.104063 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:53.104040 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:17:54.030104 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:54.030072 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh"] Oct 30 18:17:54.041723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:54.041693 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-70d58ae8-rvtzh"] Oct 30 18:17:54.804744 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:54.804711 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="546de000-9c73-49b8-8ff4-c090eb595d82" path="/var/lib/kubelet/pods/546de000-9c73-49b8-8ff4-c090eb595d82/volumes" Oct 30 18:17:57.168457 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:17:57.168418 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:17:58.872756 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:58.872725 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5"] Oct 30 18:17:58.874398 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:58.874370 5232 cpu_manager.go:414] "containerMap: removing container" podUID="6e407db5-83a8-4496-843e-0974196bd4d7" containerName="revoke-keys" Oct 30 18:17:58.874398 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:58.874397 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e407db5-83a8-4496-843e-0974196bd4d7" containerName="revoke-keys" Oct 30 18:17:58.880687 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:58.880660 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e407db5-83a8-4496-843e-0974196bd4d7" containerName="revoke-keys" Oct 30 18:17:58.895007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:58.894970 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5" Oct 30 18:17:58.895999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:58.895978 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5"] Oct 30 18:17:58.911998 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:58.911967 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:17:58.913946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:58.913915 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65aa4380-jz8g5\" (UID: \"1157f8e6-65df-4a9e-9df2-d11a0eb1825e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5" Oct 30 18:17:58.914357 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:58.914325 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smppt\" (UniqueName: \"kubernetes.io/projected/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-kube-api-access-smppt\") pod \"maas-api-revoke-sub-65aa4380-jz8g5\" (UID: \"1157f8e6-65df-4a9e-9df2-d11a0eb1825e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5" Oct 30 18:17:58.917792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:58.917768 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65aa4380-jz8g5\" (UID: \"1157f8e6-65df-4a9e-9df2-d11a0eb1825e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5" Oct 30 18:17:58.923421 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:58.923395 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:17:59.019179 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:59.019144 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65aa4380-jz8g5\" (UID: \"1157f8e6-65df-4a9e-9df2-d11a0eb1825e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5" Oct 30 18:17:59.019341 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:59.019240 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65aa4380-jz8g5\" (UID: \"1157f8e6-65df-4a9e-9df2-d11a0eb1825e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5" Oct 30 18:17:59.019416 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:59.019400 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smppt\" (UniqueName: \"kubernetes.io/projected/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-kube-api-access-smppt\") pod \"maas-api-revoke-sub-65aa4380-jz8g5\" (UID: \"1157f8e6-65df-4a9e-9df2-d11a0eb1825e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5" Oct 30 18:17:59.019913 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:59.019884 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-65aa4380-jz8g5\" (UID: \"1157f8e6-65df-4a9e-9df2-d11a0eb1825e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5" Oct 30 18:17:59.021906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:59.021878 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-65aa4380-jz8g5\" (UID: \"1157f8e6-65df-4a9e-9df2-d11a0eb1825e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5" Oct 30 18:17:59.038486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:59.038463 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smppt\" (UniqueName: \"kubernetes.io/projected/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-kube-api-access-smppt\") pod \"maas-api-revoke-sub-65aa4380-jz8g5\" (UID: \"1157f8e6-65df-4a9e-9df2-d11a0eb1825e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5" Oct 30 18:17:59.221135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:59.221051 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5" Oct 30 18:17:59.536059 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:17:59.536022 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1157f8e6_65df_4a9e_9df2_d11a0eb1825e.slice/crio-0e96ed581e448b521c3b10d78cccbc219a14820704957788425afa96a30ed69a WatchSource:0}: Error finding container 0e96ed581e448b521c3b10d78cccbc219a14820704957788425afa96a30ed69a: Status 404 returned error can't find the container with id 0e96ed581e448b521c3b10d78cccbc219a14820704957788425afa96a30ed69a Oct 30 18:17:59.536322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:17:59.536301 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5"] Oct 30 18:18:00.277173 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:00.277125 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5" event={"ID":"1157f8e6-65df-4a9e-9df2-d11a0eb1825e","Type":"ContainerStarted","Data":"a00d6fa99fc9d539c5bde35eece222551e33dcbbd2631163ad2f05fddd002fd8"} Oct 30 18:18:00.277659 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:00.277210 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5" event={"ID":"1157f8e6-65df-4a9e-9df2-d11a0eb1825e","Type":"ContainerStarted","Data":"0e96ed581e448b521c3b10d78cccbc219a14820704957788425afa96a30ed69a"} Oct 30 18:18:00.320069 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:00.320011 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5" podStartSLOduration=2.319995379 podStartE2EDuration="2.319995379s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:17:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:00.316280474 +0000 UTC m=+979.632700794" watchObservedRunningTime="2026-10-30 18:18:00.319995379 +0000 UTC m=+979.636415730" Oct 30 18:18:01.288485 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:01.288446 5232 generic.go:356] "Generic (PLEG): container finished" podID="1157f8e6-65df-4a9e-9df2-d11a0eb1825e" containerID="a00d6fa99fc9d539c5bde35eece222551e33dcbbd2631163ad2f05fddd002fd8" exitCode=0 Oct 30 18:18:01.288898 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:01.288534 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5" event={"ID":"1157f8e6-65df-4a9e-9df2-d11a0eb1825e","Type":"ContainerDied","Data":"a00d6fa99fc9d539c5bde35eece222551e33dcbbd2631163ad2f05fddd002fd8"} Oct 30 18:18:02.055536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:02.054566 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz"] Oct 30 18:18:02.099551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:02.097165 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m"] Oct 30 18:18:02.143492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:02.143429 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7421d87b-hb5rz"] Oct 30 18:18:02.190462 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:02.189670 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-57a50f8c-f2m5m"] Oct 30 18:18:02.729578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:02.729462 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx"] Oct 30 18:18:02.759852 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:02.759817 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx" Oct 30 18:18:02.795796 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:02.792254 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-7cb9c50e\"" Oct 30 18:18:02.795796 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:02.792403 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-7cb9c50e-dockercfg-cxczq\"" Oct 30 18:18:02.882270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:02.850908 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00393083-9ee4-4a04-90bd-55e9de66d5e7-plugins-config-volume\") pod \"payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx\" (UID: \"00393083-9ee4-4a04-90bd-55e9de66d5e7\") " pod="openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx" Oct 30 18:18:02.882270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:02.851097 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pbvq6\" (UniqueName: \"kubernetes.io/projected/00393083-9ee4-4a04-90bd-55e9de66d5e7-kube-api-access-pbvq6\") pod \"payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx\" (UID: \"00393083-9ee4-4a04-90bd-55e9de66d5e7\") " pod="openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx" Oct 30 18:18:02.921038 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:02.920946 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="757d4a71-96c1-4d34-a8a4-2e0cf6328e18" path="/var/lib/kubelet/pods/757d4a71-96c1-4d34-a8a4-2e0cf6328e18/volumes" Oct 30 18:18:02.922297 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:02.922113 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a77ef0ef-32af-46d9-86e5-8fc577144050" path="/var/lib/kubelet/pods/a77ef0ef-32af-46d9-86e5-8fc577144050/volumes" Oct 30 18:18:02.922984 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:02.922964 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx"] Oct 30 18:18:02.952313 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:02.952264 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pbvq6\" (UniqueName: \"kubernetes.io/projected/00393083-9ee4-4a04-90bd-55e9de66d5e7-kube-api-access-pbvq6\") pod \"payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx\" (UID: \"00393083-9ee4-4a04-90bd-55e9de66d5e7\") " pod="openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx" Oct 30 18:18:02.952456 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:02.952376 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00393083-9ee4-4a04-90bd-55e9de66d5e7-plugins-config-volume\") pod \"payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx\" (UID: \"00393083-9ee4-4a04-90bd-55e9de66d5e7\") " pod="openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx" Oct 30 18:18:02.953100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:02.953081 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00393083-9ee4-4a04-90bd-55e9de66d5e7-plugins-config-volume\") pod \"payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx\" (UID: \"00393083-9ee4-4a04-90bd-55e9de66d5e7\") " pod="openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx" Oct 30 18:18:02.973675 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:02.973307 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pbvq6\" (UniqueName: \"kubernetes.io/projected/00393083-9ee4-4a04-90bd-55e9de66d5e7-kube-api-access-pbvq6\") pod \"payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx\" (UID: \"00393083-9ee4-4a04-90bd-55e9de66d5e7\") " pod="openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx" Oct 30 18:18:03.104651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:03.104615 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5" Oct 30 18:18:03.160694 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:03.160666 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-kube-api-access-smppt\" (UniqueName: \"kubernetes.io/projected/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-kube-api-access-smppt\") pod \"1157f8e6-65df-4a9e-9df2-d11a0eb1825e\" (UID: \"1157f8e6-65df-4a9e-9df2-d11a0eb1825e\") " Oct 30 18:18:03.160854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:03.160728 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-maas-api-xks-ca\") pod \"1157f8e6-65df-4a9e-9df2-d11a0eb1825e\" (UID: \"1157f8e6-65df-4a9e-9df2-d11a0eb1825e\") " Oct 30 18:18:03.160854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:03.160766 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-maas-api-openshift-ca\") pod \"1157f8e6-65df-4a9e-9df2-d11a0eb1825e\" (UID: \"1157f8e6-65df-4a9e-9df2-d11a0eb1825e\") " Oct 30 18:18:03.161170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:03.161135 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-maas-api-openshift-ca" pod "1157f8e6-65df-4a9e-9df2-d11a0eb1825e" (UID: "1157f8e6-65df-4a9e-9df2-d11a0eb1825e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:03.161682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:03.161661 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:18:03.163056 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:03.163034 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-maas-api-xks-ca" pod "1157f8e6-65df-4a9e-9df2-d11a0eb1825e" (UID: "1157f8e6-65df-4a9e-9df2-d11a0eb1825e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:03.163235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:03.163208 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-kube-api-access-smppt" pod "1157f8e6-65df-4a9e-9df2-d11a0eb1825e" (UID: "1157f8e6-65df-4a9e-9df2-d11a0eb1825e"). InnerVolumeSpecName "kube-api-access-smppt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:03.247611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:03.247580 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx" Oct 30 18:18:03.263980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:03.263948 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smppt\" (UniqueName: \"kubernetes.io/projected/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-kube-api-access-smppt\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:18:03.263980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:03.263979 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1157f8e6-65df-4a9e-9df2-d11a0eb1825e-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:18:03.379030 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:03.378952 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5" Oct 30 18:18:03.379030 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:03.379011 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5" event={"ID":"1157f8e6-65df-4a9e-9df2-d11a0eb1825e","Type":"ContainerDied","Data":"0e96ed581e448b521c3b10d78cccbc219a14820704957788425afa96a30ed69a"} Oct 30 18:18:03.379215 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:03.379063 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0e96ed581e448b521c3b10d78cccbc219a14820704957788425afa96a30ed69a" Oct 30 18:18:03.693598 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:03.691628 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx"] Oct 30 18:18:04.668588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:04.665945 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx" event={"ID":"00393083-9ee4-4a04-90bd-55e9de66d5e7","Type":"ContainerStarted","Data":"42b707962a3cc5221cdc1ee06e9c217e011c8e06f33d0ebb1eb36b849f8928fd"} Oct 30 18:18:04.668588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:04.666048 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx" event={"ID":"00393083-9ee4-4a04-90bd-55e9de66d5e7","Type":"ContainerStarted","Data":"5083c69588017a9017b8162bbb2e8127667c4a15a7f6fcc582822f23e2665f0e"} Oct 30 18:18:04.668588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:04.667603 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx" Oct 30 18:18:04.782535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:04.780845 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx" podStartSLOduration=2.780822332 podStartE2EDuration="2.780822332s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:04.746737669 +0000 UTC m=+984.063157992" watchObservedRunningTime="2026-10-30 18:18:04.780822332 +0000 UTC m=+984.097242654" Oct 30 18:18:05.365241 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:05.364092 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:18:05.366782 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:05.366751 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:18:05.557911 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:05.557866 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:05.576100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:05.576058 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:05.630698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:05.630602 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:18:05.766670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:05.766623 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:18:05.777608 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:05.776612 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:05.860012 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:05.859965 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:05.861597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:05.860242 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:05.967156 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:05.967054 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.076561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.067733 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww"] Oct 30 18:18:06.215329 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.215275 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-383e3cd8-trkww"] Oct 30 18:18:06.366675 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.366626 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.367621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.367598 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:18:06.385771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.369560 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.387783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.387757 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:18:06.447851 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.447813 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.582583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.571057 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.615609 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.584197 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:18:06.619721 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.619625 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:18:06.640678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.638269 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:18:06.640678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.640202 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:18:06.665544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.641254 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:18:06.801579 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.783848 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:18:06.801579 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.784354 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:18:06.857574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.849325 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.888641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.878326 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.891442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.888907 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:06.980540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:06.970962 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.153006 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:07.152916 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:07.153280 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:07.153249 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" podUID="31f08f06-6d9f-4003-b35a-62de857fbe80" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.104:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.176813 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:07.176741 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e06f351d-6504-44b5-a76b-90d9eaff99b2" path="/var/lib/kubelet/pods/e06f351d-6504-44b5-a76b-90d9eaff99b2/volumes" Oct 30 18:18:07.203427 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:07.203382 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:18:07.203834 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:07.203814 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:18:07.204070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:07.204038 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:18:07.204307 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:07.204236 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:18:07.370060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:07.370016 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.405751 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:07.405640 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.458319 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:07.458275 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.470700 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:07.470656 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:18:07.554348 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:07.552688 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Oct 30 18:18:07.597554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:07.592283 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:18:07.597554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:07.594242 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.645588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:07.637398 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:18:07.645588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:07.640376 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.797533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:07.797151 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:07.878537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:07.850634 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 18:18:08.166090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:08.165285 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:18:08.270539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:08.269878 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:08.298547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:08.271956 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:08.298547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:08.283603 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:18:08.626641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:08.396627 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:08.626641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:08.491965 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:08.626641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:08.519734 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:08.712813 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:08.712771 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:08.811533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:08.811094 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:08.940598 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:08.915234 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:09.310878 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:09.310828 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:09.312484 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:08.949161 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:09.313284 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:09.130635 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:09.320325 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:09.155782 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": context deadline exceeded" Oct 30 18:18:09.324529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:09.243743 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:18:09.369549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:09.355248 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:09.369784 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:09.369723 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:09.421537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:09.396829 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:09.627599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:09.492204 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:09.627599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:09.521609 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:18:09.627599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:09.593610 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:09.627599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:09.593692 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" podUID="3480923d-8625-430e-819c-628e203c4946" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.30:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:09.627599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:09.605816 5232 patch_prober.go:29] interesting pod/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:09.627599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:09.605891 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" podUID="f8c0e348-2b8d-48e2-9ec0-a21a83db391e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.31:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:09.712120 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:09.672911 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:09.712120 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:09.673052 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" podUID="693cdf78-86cc-4f53-9c17-3926b662aeb4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": context deadline exceeded" Oct 30 18:18:09.740755 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:09.740699 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:09.885590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:09.863224 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:18:10.089750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:10.089700 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:10.133294 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:10.133248 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded" Oct 30 18:18:10.188278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:10.188184 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:10.284097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:10.284038 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:18:10.417031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:10.416976 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:18:10.684646 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:10.417303 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:18:10.759305 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:10.417377 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:18:10.763061 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:10.417402 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:10.774727 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:10.437386 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:18:10.780565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:10.530369 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:18:10.832593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:10.531673 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:10.832593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:10.662630 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:10.916796 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:10.916755 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:18:10.918003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:10.917968 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:10.985594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:10.984223 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:11.175214 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.175180 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:11.175876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.175814 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" podUID="b090d280-3716-4098-9330-7fd924785b9d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.119:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:11.249628 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.249541 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:18:11.357256 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.346681 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:18:11.443324 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.439754 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:18:11.507749 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.507676 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:18:11.513588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.513565 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:18:11.515224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.515208 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:18:11.527014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.526988 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:18:11.568531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.545449 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:18:11.594040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.593789 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:18:11.595851 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.595795 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:18:11.616441 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.616414 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f"] Oct 30 18:18:11.618997 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.618971 5232 cpu_manager.go:414] "containerMap: removing container" podUID="1157f8e6-65df-4a9e-9df2-d11a0eb1825e" containerName="revoke-keys" Oct 30 18:18:11.619531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.619493 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1157f8e6-65df-4a9e-9df2-d11a0eb1825e" containerName="revoke-keys" Oct 30 18:18:11.620333 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.620316 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="1157f8e6-65df-4a9e-9df2-d11a0eb1825e" containerName="revoke-keys" Oct 30 18:18:11.627566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.626120 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:18:11.627566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.626153 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f"] Oct 30 18:18:11.627566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.627066 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f" Oct 30 18:18:11.642933 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.642038 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:18:11.642933 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.642151 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:18:11.793709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.791835 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwzjz\" (UniqueName: \"kubernetes.io/projected/1d08e350-a49a-444e-b60b-0d1b97680381-kube-api-access-nwzjz\") pod \"maas-api-revoke-sub-aee7087a-kkz2f\" (UID: \"1d08e350-a49a-444e-b60b-0d1b97680381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f" Oct 30 18:18:11.793709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.791891 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d08e350-a49a-444e-b60b-0d1b97680381-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aee7087a-kkz2f\" (UID: \"1d08e350-a49a-444e-b60b-0d1b97680381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f" Oct 30 18:18:11.793709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.791971 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d08e350-a49a-444e-b60b-0d1b97680381-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aee7087a-kkz2f\" (UID: \"1d08e350-a49a-444e-b60b-0d1b97680381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f" Oct 30 18:18:11.893815 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.893773 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d08e350-a49a-444e-b60b-0d1b97680381-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aee7087a-kkz2f\" (UID: \"1d08e350-a49a-444e-b60b-0d1b97680381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f" Oct 30 18:18:11.894662 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.894631 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d08e350-a49a-444e-b60b-0d1b97680381-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aee7087a-kkz2f\" (UID: \"1d08e350-a49a-444e-b60b-0d1b97680381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f" Oct 30 18:18:11.895108 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.895081 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwzjz\" (UniqueName: \"kubernetes.io/projected/1d08e350-a49a-444e-b60b-0d1b97680381-kube-api-access-nwzjz\") pod \"maas-api-revoke-sub-aee7087a-kkz2f\" (UID: \"1d08e350-a49a-444e-b60b-0d1b97680381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f" Oct 30 18:18:11.895324 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.895129 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d08e350-a49a-444e-b60b-0d1b97680381-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aee7087a-kkz2f\" (UID: \"1d08e350-a49a-444e-b60b-0d1b97680381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f" Oct 30 18:18:11.901928 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.901903 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d08e350-a49a-444e-b60b-0d1b97680381-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aee7087a-kkz2f\" (UID: \"1d08e350-a49a-444e-b60b-0d1b97680381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f" Oct 30 18:18:11.921658 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.921628 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwzjz\" (UniqueName: \"kubernetes.io/projected/1d08e350-a49a-444e-b60b-0d1b97680381-kube-api-access-nwzjz\") pod \"maas-api-revoke-sub-aee7087a-kkz2f\" (UID: \"1d08e350-a49a-444e-b60b-0d1b97680381\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f" Oct 30 18:18:11.948185 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.948150 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8"] Oct 30 18:18:11.950422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.950391 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f" Oct 30 18:18:11.954241 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.953813 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8" Oct 30 18:18:11.987532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:11.983060 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8"] Oct 30 18:18:12.120534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:12.117499 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddfc42c9-0e07-4789-8197-e70ca31c4769-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39227f26-7k4v8\" (UID: \"ddfc42c9-0e07-4789-8197-e70ca31c4769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8" Oct 30 18:18:12.120534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:12.117595 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddfc42c9-0e07-4789-8197-e70ca31c4769-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39227f26-7k4v8\" (UID: \"ddfc42c9-0e07-4789-8197-e70ca31c4769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8" Oct 30 18:18:12.120534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:12.117759 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chvls\" (UniqueName: \"kubernetes.io/projected/ddfc42c9-0e07-4789-8197-e70ca31c4769-kube-api-access-chvls\") pod \"maas-api-revoke-sub-39227f26-7k4v8\" (UID: \"ddfc42c9-0e07-4789-8197-e70ca31c4769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8" Oct 30 18:18:12.230070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:12.229927 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chvls\" (UniqueName: \"kubernetes.io/projected/ddfc42c9-0e07-4789-8197-e70ca31c4769-kube-api-access-chvls\") pod \"maas-api-revoke-sub-39227f26-7k4v8\" (UID: \"ddfc42c9-0e07-4789-8197-e70ca31c4769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8" Oct 30 18:18:12.230578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:12.230189 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddfc42c9-0e07-4789-8197-e70ca31c4769-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39227f26-7k4v8\" (UID: \"ddfc42c9-0e07-4789-8197-e70ca31c4769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8" Oct 30 18:18:12.230578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:12.230240 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddfc42c9-0e07-4789-8197-e70ca31c4769-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39227f26-7k4v8\" (UID: \"ddfc42c9-0e07-4789-8197-e70ca31c4769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8" Oct 30 18:18:12.298701 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:12.260196 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddfc42c9-0e07-4789-8197-e70ca31c4769-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-39227f26-7k4v8\" (UID: \"ddfc42c9-0e07-4789-8197-e70ca31c4769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8" Oct 30 18:18:12.298701 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:12.273966 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddfc42c9-0e07-4789-8197-e70ca31c4769-maas-api-xks-ca\") pod \"maas-api-revoke-sub-39227f26-7k4v8\" (UID: \"ddfc42c9-0e07-4789-8197-e70ca31c4769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8" Oct 30 18:18:12.343605 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:12.334451 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chvls\" (UniqueName: \"kubernetes.io/projected/ddfc42c9-0e07-4789-8197-e70ca31c4769-kube-api-access-chvls\") pod \"maas-api-revoke-sub-39227f26-7k4v8\" (UID: \"ddfc42c9-0e07-4789-8197-e70ca31c4769\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8" Oct 30 18:18:12.575005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:12.574974 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f"] Oct 30 18:18:12.613563 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:12.613527 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8" Oct 30 18:18:12.933611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:12.933497 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f" event={"ID":"1d08e350-a49a-444e-b60b-0d1b97680381","Type":"ContainerStarted","Data":"98d10f03f5963f2d3d4a92b1c916bc29a2927ea985d308b92d47cc011447f9a4"} Oct 30 18:18:13.008949 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:18:13.008916 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddfc42c9_0e07_4789_8197_e70ca31c4769.slice/crio-ed4586840520b9f7c2a20a54c56cc79ed35b5981b2ee980add2a1df5f8708a2b WatchSource:0}: Error finding container ed4586840520b9f7c2a20a54c56cc79ed35b5981b2ee980add2a1df5f8708a2b: Status 404 returned error can't find the container with id ed4586840520b9f7c2a20a54c56cc79ed35b5981b2ee980add2a1df5f8708a2b Oct 30 18:18:13.009109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:13.009089 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8"] Oct 30 18:18:13.945942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:13.945909 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8" event={"ID":"ddfc42c9-0e07-4789-8197-e70ca31c4769","Type":"ContainerStarted","Data":"5cef4ca50a5e450fc0cf4a4fb2f901fb6e8dc99c6897bfaee0842d673b94e425"} Oct 30 18:18:13.946407 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:13.945965 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8" event={"ID":"ddfc42c9-0e07-4789-8197-e70ca31c4769","Type":"ContainerStarted","Data":"ed4586840520b9f7c2a20a54c56cc79ed35b5981b2ee980add2a1df5f8708a2b"} Oct 30 18:18:13.947666 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:13.947640 5232 generic.go:356] "Generic (PLEG): container finished" podID="1d08e350-a49a-444e-b60b-0d1b97680381" containerID="4183454e4799771070fcaa68537ac7ec2a27f0e7b8d16d5db074aaae84037aa6" exitCode=0 Oct 30 18:18:13.947809 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:13.947725 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f" event={"ID":"1d08e350-a49a-444e-b60b-0d1b97680381","Type":"ContainerDied","Data":"4183454e4799771070fcaa68537ac7ec2a27f0e7b8d16d5db074aaae84037aa6"} Oct 30 18:18:13.984848 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:13.984769 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8" podStartSLOduration=2.9847564159999997 podStartE2EDuration="2.984756416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:13.983708694 +0000 UTC m=+993.300129016" watchObservedRunningTime="2026-10-30 18:18:13.984756416 +0000 UTC m=+993.301176736" Oct 30 18:18:14.901168 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:14.901135 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p"] Oct 30 18:18:14.905612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:14.905593 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p" Oct 30 18:18:14.923382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:14.923341 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p"] Oct 30 18:18:14.960832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:14.960801 5232 generic.go:356] "Generic (PLEG): container finished" podID="ddfc42c9-0e07-4789-8197-e70ca31c4769" containerID="5cef4ca50a5e450fc0cf4a4fb2f901fb6e8dc99c6897bfaee0842d673b94e425" exitCode=0 Oct 30 18:18:14.961247 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:14.960882 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8" event={"ID":"ddfc42c9-0e07-4789-8197-e70ca31c4769","Type":"ContainerDied","Data":"5cef4ca50a5e450fc0cf4a4fb2f901fb6e8dc99c6897bfaee0842d673b94e425"} Oct 30 18:18:15.018946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.018912 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13de057-8d57-4792-a002-990d43df53d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-516db80f-ghd9p\" (UID: \"a13de057-8d57-4792-a002-990d43df53d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p" Oct 30 18:18:15.019141 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.018976 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvqt9\" (UniqueName: \"kubernetes.io/projected/a13de057-8d57-4792-a002-990d43df53d5-kube-api-access-wvqt9\") pod \"maas-api-revoke-sub-516db80f-ghd9p\" (UID: \"a13de057-8d57-4792-a002-990d43df53d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p" Oct 30 18:18:15.019567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.019206 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13de057-8d57-4792-a002-990d43df53d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-516db80f-ghd9p\" (UID: \"a13de057-8d57-4792-a002-990d43df53d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p" Oct 30 18:18:15.120566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.120523 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13de057-8d57-4792-a002-990d43df53d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-516db80f-ghd9p\" (UID: \"a13de057-8d57-4792-a002-990d43df53d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p" Oct 30 18:18:15.120915 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.120892 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13de057-8d57-4792-a002-990d43df53d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-516db80f-ghd9p\" (UID: \"a13de057-8d57-4792-a002-990d43df53d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p" Oct 30 18:18:15.121006 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.120958 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvqt9\" (UniqueName: \"kubernetes.io/projected/a13de057-8d57-4792-a002-990d43df53d5-kube-api-access-wvqt9\") pod \"maas-api-revoke-sub-516db80f-ghd9p\" (UID: \"a13de057-8d57-4792-a002-990d43df53d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p" Oct 30 18:18:15.121917 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.121880 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13de057-8d57-4792-a002-990d43df53d5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-516db80f-ghd9p\" (UID: \"a13de057-8d57-4792-a002-990d43df53d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p" Oct 30 18:18:15.123131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.123109 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13de057-8d57-4792-a002-990d43df53d5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-516db80f-ghd9p\" (UID: \"a13de057-8d57-4792-a002-990d43df53d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p" Oct 30 18:18:15.139656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.139633 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvqt9\" (UniqueName: \"kubernetes.io/projected/a13de057-8d57-4792-a002-990d43df53d5-kube-api-access-wvqt9\") pod \"maas-api-revoke-sub-516db80f-ghd9p\" (UID: \"a13de057-8d57-4792-a002-990d43df53d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p" Oct 30 18:18:15.158578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.158501 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f" Oct 30 18:18:15.220745 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.220709 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p" Oct 30 18:18:15.324077 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.324049 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d08e350-a49a-444e-b60b-0d1b97680381-kube-api-access-nwzjz\" (UniqueName: \"kubernetes.io/projected/1d08e350-a49a-444e-b60b-0d1b97680381-kube-api-access-nwzjz\") pod \"1d08e350-a49a-444e-b60b-0d1b97680381\" (UID: \"1d08e350-a49a-444e-b60b-0d1b97680381\") " Oct 30 18:18:15.324243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.324144 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d08e350-a49a-444e-b60b-0d1b97680381-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d08e350-a49a-444e-b60b-0d1b97680381-maas-api-openshift-ca\") pod \"1d08e350-a49a-444e-b60b-0d1b97680381\" (UID: \"1d08e350-a49a-444e-b60b-0d1b97680381\") " Oct 30 18:18:15.324569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.324384 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d08e350-a49a-444e-b60b-0d1b97680381-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d08e350-a49a-444e-b60b-0d1b97680381-maas-api-xks-ca\") pod \"1d08e350-a49a-444e-b60b-0d1b97680381\" (UID: \"1d08e350-a49a-444e-b60b-0d1b97680381\") " Oct 30 18:18:15.324688 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.324641 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d08e350-a49a-444e-b60b-0d1b97680381-maas-api-openshift-ca" pod "1d08e350-a49a-444e-b60b-0d1b97680381" (UID: "1d08e350-a49a-444e-b60b-0d1b97680381"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:15.325560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.325373 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d08e350-a49a-444e-b60b-0d1b97680381-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:18:15.326792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.326768 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d08e350-a49a-444e-b60b-0d1b97680381-maas-api-xks-ca" pod "1d08e350-a49a-444e-b60b-0d1b97680381" (UID: "1d08e350-a49a-444e-b60b-0d1b97680381"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:15.326945 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.326920 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d08e350-a49a-444e-b60b-0d1b97680381-kube-api-access-nwzjz" pod "1d08e350-a49a-444e-b60b-0d1b97680381" (UID: "1d08e350-a49a-444e-b60b-0d1b97680381"). InnerVolumeSpecName "kube-api-access-nwzjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:15.449621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.445462 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d08e350-a49a-444e-b60b-0d1b97680381-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:18:15.449621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.445502 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwzjz\" (UniqueName: \"kubernetes.io/projected/1d08e350-a49a-444e-b60b-0d1b97680381-kube-api-access-nwzjz\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:18:15.564102 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:18:15.564064 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda13de057_8d57_4792_a002_990d43df53d5.slice/crio-ce7a007bd9018c3ef45168379d1d02f90ef2c25ea00324626cd186179b6a3f6a WatchSource:0}: Error finding container ce7a007bd9018c3ef45168379d1d02f90ef2c25ea00324626cd186179b6a3f6a: Status 404 returned error can't find the container with id ce7a007bd9018c3ef45168379d1d02f90ef2c25ea00324626cd186179b6a3f6a Oct 30 18:18:15.570607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.570577 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p"] Oct 30 18:18:15.971472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.971433 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p" event={"ID":"a13de057-8d57-4792-a002-990d43df53d5","Type":"ContainerStarted","Data":"ce7a007bd9018c3ef45168379d1d02f90ef2c25ea00324626cd186179b6a3f6a"} Oct 30 18:18:15.972807 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.972787 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f" Oct 30 18:18:15.972919 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.972807 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f" event={"ID":"1d08e350-a49a-444e-b60b-0d1b97680381","Type":"ContainerDied","Data":"98d10f03f5963f2d3d4a92b1c916bc29a2927ea985d308b92d47cc011447f9a4"} Oct 30 18:18:15.972919 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:15.972855 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98d10f03f5963f2d3d4a92b1c916bc29a2927ea985d308b92d47cc011447f9a4" Oct 30 18:18:16.151658 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:16.151631 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8" Oct 30 18:18:16.257398 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:16.257370 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddfc42c9-0e07-4789-8197-e70ca31c4769-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddfc42c9-0e07-4789-8197-e70ca31c4769-maas-api-openshift-ca\") pod \"ddfc42c9-0e07-4789-8197-e70ca31c4769\" (UID: \"ddfc42c9-0e07-4789-8197-e70ca31c4769\") " Oct 30 18:18:16.257575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:16.257431 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddfc42c9-0e07-4789-8197-e70ca31c4769-kube-api-access-chvls\" (UniqueName: \"kubernetes.io/projected/ddfc42c9-0e07-4789-8197-e70ca31c4769-kube-api-access-chvls\") pod \"ddfc42c9-0e07-4789-8197-e70ca31c4769\" (UID: \"ddfc42c9-0e07-4789-8197-e70ca31c4769\") " Oct 30 18:18:16.257680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:16.257661 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddfc42c9-0e07-4789-8197-e70ca31c4769-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddfc42c9-0e07-4789-8197-e70ca31c4769-maas-api-xks-ca\") pod \"ddfc42c9-0e07-4789-8197-e70ca31c4769\" (UID: \"ddfc42c9-0e07-4789-8197-e70ca31c4769\") " Oct 30 18:18:16.257739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:16.257713 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddfc42c9-0e07-4789-8197-e70ca31c4769-maas-api-openshift-ca" pod "ddfc42c9-0e07-4789-8197-e70ca31c4769" (UID: "ddfc42c9-0e07-4789-8197-e70ca31c4769"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:16.258324 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:16.258306 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddfc42c9-0e07-4789-8197-e70ca31c4769-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:18:16.259829 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:16.259804 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddfc42c9-0e07-4789-8197-e70ca31c4769-maas-api-xks-ca" pod "ddfc42c9-0e07-4789-8197-e70ca31c4769" (UID: "ddfc42c9-0e07-4789-8197-e70ca31c4769"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:16.259917 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:16.259838 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddfc42c9-0e07-4789-8197-e70ca31c4769-kube-api-access-chvls" pod "ddfc42c9-0e07-4789-8197-e70ca31c4769" (UID: "ddfc42c9-0e07-4789-8197-e70ca31c4769"). InnerVolumeSpecName "kube-api-access-chvls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:16.360314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:16.360281 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chvls\" (UniqueName: \"kubernetes.io/projected/ddfc42c9-0e07-4789-8197-e70ca31c4769-kube-api-access-chvls\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:18:16.360314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:16.360310 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddfc42c9-0e07-4789-8197-e70ca31c4769-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:18:16.983867 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:16.983766 5232 generic.go:356] "Generic (PLEG): container finished" podID="a13de057-8d57-4792-a002-990d43df53d5" containerID="c34fb0a5e29c945f8483f371d36a77529185e0cb36da2391615176a1c11afcd3" exitCode=0 Oct 30 18:18:16.983867 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:16.983809 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p" event={"ID":"a13de057-8d57-4792-a002-990d43df53d5","Type":"ContainerDied","Data":"c34fb0a5e29c945f8483f371d36a77529185e0cb36da2391615176a1c11afcd3"} Oct 30 18:18:16.985581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:16.985555 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8" Oct 30 18:18:16.985581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:16.985558 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8" event={"ID":"ddfc42c9-0e07-4789-8197-e70ca31c4769","Type":"ContainerDied","Data":"ed4586840520b9f7c2a20a54c56cc79ed35b5981b2ee980add2a1df5f8708a2b"} Oct 30 18:18:16.985771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:16.985600 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ed4586840520b9f7c2a20a54c56cc79ed35b5981b2ee980add2a1df5f8708a2b" Oct 30 18:18:17.342299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:17.342274 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx" Oct 30 18:18:18.267359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:18.267334 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p" Oct 30 18:18:18.342070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:18.341981 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a13de057-8d57-4792-a002-990d43df53d5-kube-api-access-wvqt9\" (UniqueName: \"kubernetes.io/projected/a13de057-8d57-4792-a002-990d43df53d5-kube-api-access-wvqt9\") pod \"a13de057-8d57-4792-a002-990d43df53d5\" (UID: \"a13de057-8d57-4792-a002-990d43df53d5\") " Oct 30 18:18:18.342249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:18.342143 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a13de057-8d57-4792-a002-990d43df53d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13de057-8d57-4792-a002-990d43df53d5-maas-api-xks-ca\") pod \"a13de057-8d57-4792-a002-990d43df53d5\" (UID: \"a13de057-8d57-4792-a002-990d43df53d5\") " Oct 30 18:18:18.342309 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:18.342259 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a13de057-8d57-4792-a002-990d43df53d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13de057-8d57-4792-a002-990d43df53d5-maas-api-openshift-ca\") pod \"a13de057-8d57-4792-a002-990d43df53d5\" (UID: \"a13de057-8d57-4792-a002-990d43df53d5\") " Oct 30 18:18:18.342867 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:18.342829 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a13de057-8d57-4792-a002-990d43df53d5-maas-api-openshift-ca" pod "a13de057-8d57-4792-a002-990d43df53d5" (UID: "a13de057-8d57-4792-a002-990d43df53d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:18.343138 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:18.343112 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a13de057-8d57-4792-a002-990d43df53d5-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:18:18.345252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:18.345221 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a13de057-8d57-4792-a002-990d43df53d5-maas-api-xks-ca" pod "a13de057-8d57-4792-a002-990d43df53d5" (UID: "a13de057-8d57-4792-a002-990d43df53d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:18.345398 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:18.345365 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a13de057-8d57-4792-a002-990d43df53d5-kube-api-access-wvqt9" pod "a13de057-8d57-4792-a002-990d43df53d5" (UID: "a13de057-8d57-4792-a002-990d43df53d5"). InnerVolumeSpecName "kube-api-access-wvqt9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:18.444158 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:18.444124 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvqt9\" (UniqueName: \"kubernetes.io/projected/a13de057-8d57-4792-a002-990d43df53d5-kube-api-access-wvqt9\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:18:18.444158 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:18.444154 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a13de057-8d57-4792-a002-990d43df53d5-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:18:19.045849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:19.045819 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p" Oct 30 18:18:19.046032 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:19.045847 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p" event={"ID":"a13de057-8d57-4792-a002-990d43df53d5","Type":"ContainerDied","Data":"ce7a007bd9018c3ef45168379d1d02f90ef2c25ea00324626cd186179b6a3f6a"} Oct 30 18:18:19.046032 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:19.045895 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ce7a007bd9018c3ef45168379d1d02f90ef2c25ea00324626cd186179b6a3f6a" Oct 30 18:18:22.377575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.377420 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx"] Oct 30 18:18:22.378161 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.378142 5232 cpu_manager.go:414] "containerMap: removing container" podUID="ddfc42c9-0e07-4789-8197-e70ca31c4769" containerName="revoke-keys" Oct 30 18:18:22.378207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.378164 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddfc42c9-0e07-4789-8197-e70ca31c4769" containerName="revoke-keys" Oct 30 18:18:22.378207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.378199 5232 cpu_manager.go:414] "containerMap: removing container" podUID="a13de057-8d57-4792-a002-990d43df53d5" containerName="revoke-keys" Oct 30 18:18:22.378207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.378204 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a13de057-8d57-4792-a002-990d43df53d5" containerName="revoke-keys" Oct 30 18:18:22.378314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.378225 5232 cpu_manager.go:414] "containerMap: removing container" podUID="1d08e350-a49a-444e-b60b-0d1b97680381" containerName="revoke-keys" Oct 30 18:18:22.378314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.378231 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d08e350-a49a-444e-b60b-0d1b97680381" containerName="revoke-keys" Oct 30 18:18:22.378379 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.378330 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddfc42c9-0e07-4789-8197-e70ca31c4769" containerName="revoke-keys" Oct 30 18:18:22.378379 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.378349 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="a13de057-8d57-4792-a002-990d43df53d5" containerName="revoke-keys" Oct 30 18:18:22.378379 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.378361 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d08e350-a49a-444e-b60b-0d1b97680381" containerName="revoke-keys" Oct 30 18:18:22.382709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.382691 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx" Oct 30 18:18:22.399088 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.399061 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx"] Oct 30 18:18:22.404334 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.404306 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:18:22.404494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.404374 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:18:22.428670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.428628 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb2fb1d-093e-420b-9b0c-3fe66306155d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b76989c0-pvmxx\" (UID: \"ddb2fb1d-093e-420b-9b0c-3fe66306155d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx" Oct 30 18:18:22.428856 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.428833 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tww2\" (UniqueName: \"kubernetes.io/projected/ddb2fb1d-093e-420b-9b0c-3fe66306155d-kube-api-access-8tww2\") pod \"maas-api-revoke-sub-b76989c0-pvmxx\" (UID: \"ddb2fb1d-093e-420b-9b0c-3fe66306155d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx" Oct 30 18:18:22.428951 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.428879 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb2fb1d-093e-420b-9b0c-3fe66306155d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b76989c0-pvmxx\" (UID: \"ddb2fb1d-093e-420b-9b0c-3fe66306155d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx" Oct 30 18:18:22.530140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.530109 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tww2\" (UniqueName: \"kubernetes.io/projected/ddb2fb1d-093e-420b-9b0c-3fe66306155d-kube-api-access-8tww2\") pod \"maas-api-revoke-sub-b76989c0-pvmxx\" (UID: \"ddb2fb1d-093e-420b-9b0c-3fe66306155d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx" Oct 30 18:18:22.530140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.530146 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb2fb1d-093e-420b-9b0c-3fe66306155d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b76989c0-pvmxx\" (UID: \"ddb2fb1d-093e-420b-9b0c-3fe66306155d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx" Oct 30 18:18:22.530420 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.530332 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb2fb1d-093e-420b-9b0c-3fe66306155d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b76989c0-pvmxx\" (UID: \"ddb2fb1d-093e-420b-9b0c-3fe66306155d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx" Oct 30 18:18:22.531215 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.531164 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb2fb1d-093e-420b-9b0c-3fe66306155d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b76989c0-pvmxx\" (UID: \"ddb2fb1d-093e-420b-9b0c-3fe66306155d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx" Oct 30 18:18:22.533124 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.533099 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb2fb1d-093e-420b-9b0c-3fe66306155d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b76989c0-pvmxx\" (UID: \"ddb2fb1d-093e-420b-9b0c-3fe66306155d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx" Oct 30 18:18:22.550126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.550100 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tww2\" (UniqueName: \"kubernetes.io/projected/ddb2fb1d-093e-420b-9b0c-3fe66306155d-kube-api-access-8tww2\") pod \"maas-api-revoke-sub-b76989c0-pvmxx\" (UID: \"ddb2fb1d-093e-420b-9b0c-3fe66306155d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx" Oct 30 18:18:22.700498 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:22.700422 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx" Oct 30 18:18:23.002363 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:23.002331 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx"] Oct 30 18:18:23.002667 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:18:23.002644 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podddb2fb1d_093e_420b_9b0c_3fe66306155d.slice/crio-c6a89ac05ec6e7ee2ed17879ab10aad100f26d1f8ca6f851352f41013b4fb440 WatchSource:0}: Error finding container c6a89ac05ec6e7ee2ed17879ab10aad100f26d1f8ca6f851352f41013b4fb440: Status 404 returned error can't find the container with id c6a89ac05ec6e7ee2ed17879ab10aad100f26d1f8ca6f851352f41013b4fb440 Oct 30 18:18:23.082039 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:23.082002 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx" event={"ID":"ddb2fb1d-093e-420b-9b0c-3fe66306155d","Type":"ContainerStarted","Data":"c6a89ac05ec6e7ee2ed17879ab10aad100f26d1f8ca6f851352f41013b4fb440"} Oct 30 18:18:24.030551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.030490 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm"] Oct 30 18:18:24.042655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.042627 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed26c08a-qkrsm"] Oct 30 18:18:24.096426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.096378 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx" event={"ID":"ddb2fb1d-093e-420b-9b0c-3fe66306155d","Type":"ContainerStarted","Data":"bc9a8cadf000d65946c2a97a655b30b0fc32927d9070aee539a27b3a2248058e"} Oct 30 18:18:24.173053 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.172989 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx" podStartSLOduration=2.172969548 podStartE2EDuration="2.172969548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:24.165296143 +0000 UTC m=+1003.481716479" watchObservedRunningTime="2026-10-30 18:18:24.172969548 +0000 UTC m=+1003.489389869" Oct 30 18:18:24.430582 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.430396 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk"] Oct 30 18:18:24.437447 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.437415 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.438837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.438807 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk"] Oct 30 18:18:24.522105 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.522065 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.522300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.522193 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.522361 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.522300 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.522414 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.522381 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jwbgk\" (UniqueName: \"kubernetes.io/projected/17ccda47-32b1-42f6-9030-0c040fc73883-kube-api-access-jwbgk\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.522470 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.522450 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.522574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.522556 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ccda47-32b1-42f6-9030-0c040fc73883-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.522632 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.522593 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ccda47-32b1-42f6-9030-0c040fc73883-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.522681 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.522633 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ccda47-32b1-42f6-9030-0c040fc73883-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.522730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.522702 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.648537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.624886 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jwbgk\" (UniqueName: \"kubernetes.io/projected/17ccda47-32b1-42f6-9030-0c040fc73883-kube-api-access-jwbgk\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.650279 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.650249 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.650609 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.650580 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ccda47-32b1-42f6-9030-0c040fc73883-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.652727 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.652702 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ccda47-32b1-42f6-9030-0c040fc73883-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.653224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.653201 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ccda47-32b1-42f6-9030-0c040fc73883-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.653413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.653394 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.653614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.653596 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.653842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.653825 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.654051 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.654034 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.673561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.652150 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ccda47-32b1-42f6-9030-0c040fc73883-istiod-ca-cert\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.675169 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.675139 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-istio-data\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.676912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.676833 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-workload-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.676912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.651101 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-credential-socket\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.677355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.677333 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-workload-certs\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.688204 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.688135 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ccda47-32b1-42f6-9030-0c040fc73883-istio-podinfo\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.689674 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.689647 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ccda47-32b1-42f6-9030-0c040fc73883-istio-token\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.707449 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.707386 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-istio-envoy\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.707794 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.707766 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jwbgk\" (UniqueName: \"kubernetes.io/projected/17ccda47-32b1-42f6-9030-0c040fc73883-kube-api-access-jwbgk\") pod \"e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.764534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.758331 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:24.853629 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:24.851679 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2322f42-c68c-4e5a-a7ae-ed6b204728b6" path="/var/lib/kubelet/pods/c2322f42-c68c-4e5a-a7ae-ed6b204728b6/volumes" Oct 30 18:18:25.117627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:25.117578 5232 generic.go:356] "Generic (PLEG): container finished" podID="ddb2fb1d-093e-420b-9b0c-3fe66306155d" containerID="bc9a8cadf000d65946c2a97a655b30b0fc32927d9070aee539a27b3a2248058e" exitCode=0 Oct 30 18:18:25.117627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:25.117637 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx" event={"ID":"ddb2fb1d-093e-420b-9b0c-3fe66306155d","Type":"ContainerDied","Data":"bc9a8cadf000d65946c2a97a655b30b0fc32927d9070aee539a27b3a2248058e"} Oct 30 18:18:25.181006 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:18:25.180952 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ccda47_32b1_42f6_9030_0c040fc73883.slice/crio-3ce841e0fa18de102057ae00c5008fbc14828978a439b2255418b864177fff26 WatchSource:0}: Error finding container 3ce841e0fa18de102057ae00c5008fbc14828978a439b2255418b864177fff26: Status 404 returned error can't find the container with id 3ce841e0fa18de102057ae00c5008fbc14828978a439b2255418b864177fff26 Oct 30 18:18:25.184757 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:25.184712 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:18:25.184869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:25.184810 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:18:25.184869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:25.184853 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:18:25.187792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:25.187768 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk"] Oct 30 18:18:26.144364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:26.144324 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" event={"ID":"17ccda47-32b1-42f6-9030-0c040fc73883","Type":"ContainerStarted","Data":"4762ad491b3bde8ce19235625990fe0e46e2ab599ccc9dbc6bb619b096b441d2"} Oct 30 18:18:26.144853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:26.144391 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" event={"ID":"17ccda47-32b1-42f6-9030-0c040fc73883","Type":"ContainerStarted","Data":"3ce841e0fa18de102057ae00c5008fbc14828978a439b2255418b864177fff26"} Oct 30 18:18:26.212634 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:26.211694 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" podStartSLOduration=2.211674731 podStartE2EDuration="2.211674731s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:26.197244588 +0000 UTC m=+1005.513664934" watchObservedRunningTime="2026-10-30 18:18:26.211674731 +0000 UTC m=+1005.528095053" Oct 30 18:18:26.378425 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:26.378401 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx" Oct 30 18:18:26.530585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:26.530549 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ddb2fb1d-093e-420b-9b0c-3fe66306155d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb2fb1d-093e-420b-9b0c-3fe66306155d-maas-api-openshift-ca\") pod \"ddb2fb1d-093e-420b-9b0c-3fe66306155d\" (UID: \"ddb2fb1d-093e-420b-9b0c-3fe66306155d\") " Oct 30 18:18:26.530954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:26.530935 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ddb2fb1d-093e-420b-9b0c-3fe66306155d-kube-api-access-8tww2\" (UniqueName: \"kubernetes.io/projected/ddb2fb1d-093e-420b-9b0c-3fe66306155d-kube-api-access-8tww2\") pod \"ddb2fb1d-093e-420b-9b0c-3fe66306155d\" (UID: \"ddb2fb1d-093e-420b-9b0c-3fe66306155d\") " Oct 30 18:18:26.531148 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:26.531132 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ddb2fb1d-093e-420b-9b0c-3fe66306155d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb2fb1d-093e-420b-9b0c-3fe66306155d-maas-api-xks-ca\") pod \"ddb2fb1d-093e-420b-9b0c-3fe66306155d\" (UID: \"ddb2fb1d-093e-420b-9b0c-3fe66306155d\") " Oct 30 18:18:26.538339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:26.538304 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ddb2fb1d-093e-420b-9b0c-3fe66306155d-maas-api-xks-ca" pod "ddb2fb1d-093e-420b-9b0c-3fe66306155d" (UID: "ddb2fb1d-093e-420b-9b0c-3fe66306155d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:26.538726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:26.538695 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ddb2fb1d-093e-420b-9b0c-3fe66306155d-maas-api-openshift-ca" pod "ddb2fb1d-093e-420b-9b0c-3fe66306155d" (UID: "ddb2fb1d-093e-420b-9b0c-3fe66306155d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:26.541665 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:26.541600 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ddb2fb1d-093e-420b-9b0c-3fe66306155d-kube-api-access-8tww2" pod "ddb2fb1d-093e-420b-9b0c-3fe66306155d" (UID: "ddb2fb1d-093e-420b-9b0c-3fe66306155d"). InnerVolumeSpecName "kube-api-access-8tww2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:26.634535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:26.633861 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ddb2fb1d-093e-420b-9b0c-3fe66306155d-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:18:26.634535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:26.633898 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tww2\" (UniqueName: \"kubernetes.io/projected/ddb2fb1d-093e-420b-9b0c-3fe66306155d-kube-api-access-8tww2\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:18:26.634535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:26.633913 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ddb2fb1d-093e-420b-9b0c-3fe66306155d-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:18:26.767298 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:26.767271 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:26.768930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:26.768870 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 18:18:26.769071 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:26.768929 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" podUID="17ccda47-32b1-42f6-9030-0c040fc73883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 18:18:27.062361 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:18:27.062323 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:18:27.154721 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:27.154641 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx" Oct 30 18:18:27.154721 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:27.154684 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx" event={"ID":"ddb2fb1d-093e-420b-9b0c-3fe66306155d","Type":"ContainerDied","Data":"c6a89ac05ec6e7ee2ed17879ab10aad100f26d1f8ca6f851352f41013b4fb440"} Oct 30 18:18:27.155170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:27.154733 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c6a89ac05ec6e7ee2ed17879ab10aad100f26d1f8ca6f851352f41013b4fb440" Oct 30 18:18:27.767025 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:27.766989 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 18:18:27.767192 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:27.767060 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" podUID="17ccda47-32b1-42f6-9030-0c040fc73883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 18:18:28.767422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:28.767386 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 18:18:28.767869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:28.767453 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" podUID="17ccda47-32b1-42f6-9030-0c040fc73883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 18:18:29.044043 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:29.043971 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk"] Oct 30 18:18:29.051024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:29.050994 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-130ee7b3-tjxzk"] Oct 30 18:18:29.767004 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:29.766967 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 18:18:29.767172 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:29.767025 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" podUID="17ccda47-32b1-42f6-9030-0c040fc73883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 18:18:30.043117 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:30.042840 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk"] Oct 30 18:18:30.053215 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:30.053177 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36b1c2b3-snpjk"] Oct 30 18:18:30.767249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:30.767216 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 18:18:30.767432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:30.767275 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" podUID="17ccda47-32b1-42f6-9030-0c040fc73883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 18:18:30.804432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:30.804400 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77678ec4-a4ae-4303-9961-ff06912bf20d" path="/var/lib/kubelet/pods/77678ec4-a4ae-4303-9961-ff06912bf20d/volumes" Oct 30 18:18:30.804962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:30.804947 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9d5a28e0-197e-430e-996e-38f50ab0b723" path="/var/lib/kubelet/pods/9d5a28e0-197e-430e-996e-38f50ab0b723/volumes" Oct 30 18:18:31.863591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:31.861653 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 18:18:31.863591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:31.861728 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" podUID="17ccda47-32b1-42f6-9030-0c040fc73883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 18:18:31.963617 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:31.963581 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2"] Oct 30 18:18:32.052544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:31.968578 5232 cpu_manager.go:414] "containerMap: removing container" podUID="ddb2fb1d-093e-420b-9b0c-3fe66306155d" containerName="revoke-keys" Oct 30 18:18:32.052791 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.052764 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ddb2fb1d-093e-420b-9b0c-3fe66306155d" containerName="revoke-keys" Oct 30 18:18:32.089533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.060851 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="ddb2fb1d-093e-420b-9b0c-3fe66306155d" containerName="revoke-keys" Oct 30 18:18:32.117583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.096024 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2"] Oct 30 18:18:32.117583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.096152 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.171536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.154554 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-a90958fa-openshift-default-dockercfg-p5rmp\"" Oct 30 18:18:32.218249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.216593 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks"] Oct 30 18:18:32.255625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.220078 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-data\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.255625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.220206 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-podinfo\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.255625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.220247 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-workload-certs\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.255625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.220301 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19303a05-780a-4c03-9df6-2db3aeee3a9b-istiod-ca-cert\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.255625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.220461 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-envoy\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.255625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.220523 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-token\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.255625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.220757 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-workload-socket\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.255625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.220950 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-credential-socket\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.255625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.220982 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jjhmj\" (UniqueName: \"kubernetes.io/projected/19303a05-780a-4c03-9df6-2db3aeee3a9b-kube-api-access-jjhmj\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.272855 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.272785 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4eb098bf-ff6ks"] Oct 30 18:18:32.291061 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.289689 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:32.300647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.299137 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:18:32.349536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.322846 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19303a05-780a-4c03-9df6-2db3aeee3a9b-istiod-ca-cert\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.349536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.323001 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-envoy\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.349536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.323030 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-token\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.349536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.323121 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-workload-socket\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.349536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.323325 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-credential-socket\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.349536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.323354 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jjhmj\" (UniqueName: \"kubernetes.io/projected/19303a05-780a-4c03-9df6-2db3aeee3a9b-kube-api-access-jjhmj\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.349536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.323388 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-data\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.349536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.323457 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-podinfo\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.349536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.323496 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-workload-certs\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.349536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.323934 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-workload-certs\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.349536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.332906 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-credential-socket\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.349536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.333658 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-workload-socket\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.349536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.333943 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-data\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.349536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.335295 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19303a05-780a-4c03-9df6-2db3aeee3a9b-istiod-ca-cert\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.378557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.371815 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-envoy\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.396666 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.396619 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:32.528216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.520496 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-podinfo\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.551987 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.551957 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jjhmj\" (UniqueName: \"kubernetes.io/projected/19303a05-780a-4c03-9df6-2db3aeee3a9b-kube-api-access-jjhmj\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.552811 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.552779 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-token\") pod \"e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.577957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.577923 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:32.767096 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.767057 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" start-of-body= Oct 30 18:18:32.767273 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.767113 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" podUID="17ccda47-32b1-42f6-9030-0c040fc73883" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.133:15021/healthz/ready\": dial tcp 10.133.0.133:15021: connect: connection refused" Oct 30 18:18:32.817206 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.817159 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cabcd183-716c-4012-8a6f-054b1cf65d47" path="/var/lib/kubelet/pods/cabcd183-716c-4012-8a6f-054b1cf65d47/volumes" Oct 30 18:18:32.901234 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:18:32.901199 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod19303a05_780a_4c03_9df6_2db3aeee3a9b.slice/crio-27dce2ba098603e59827c12a839424b209eb66b245c3c06b68e0bbf6d6cbefab WatchSource:0}: Error finding container 27dce2ba098603e59827c12a839424b209eb66b245c3c06b68e0bbf6d6cbefab: Status 404 returned error can't find the container with id 27dce2ba098603e59827c12a839424b209eb66b245c3c06b68e0bbf6d6cbefab Oct 30 18:18:32.901960 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.901934 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2"] Oct 30 18:18:32.904285 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.904251 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:18:32.904373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.904317 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:18:32.904373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:32.904347 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:18:33.544995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:33.544956 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" event={"ID":"19303a05-780a-4c03-9df6-2db3aeee3a9b","Type":"ContainerStarted","Data":"437a7ca20c9e513770e6ef040e776c44c14c94542879f7b5613757bb47bbcdcf"} Oct 30 18:18:33.545175 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:33.545014 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" event={"ID":"19303a05-780a-4c03-9df6-2db3aeee3a9b","Type":"ContainerStarted","Data":"27dce2ba098603e59827c12a839424b209eb66b245c3c06b68e0bbf6d6cbefab"} Oct 30 18:18:33.578189 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:33.578159 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:33.583852 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:33.583828 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:33.594395 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:33.594351 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" podStartSLOduration=2.59433686 podStartE2EDuration="2.59433686s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:33.588275145 +0000 UTC m=+1012.904695467" watchObservedRunningTime="2026-10-30 18:18:33.59433686 +0000 UTC m=+1012.910757180" Oct 30 18:18:34.286905 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:34.286873 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:34.287618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:34.287595 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:34.288291 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:34.288274 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:18:34.555807 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:34.555719 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:34.556618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:34.556597 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:18:35.035446 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:35.035415 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t"] Oct 30 18:18:35.041354 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:35.041331 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl"] Oct 30 18:18:35.046572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:35.046551 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd235ee3-pvj2t"] Oct 30 18:18:35.051136 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:35.051117 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c0a2786d-hmrsl"] Oct 30 18:18:36.870538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:36.857694 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="056a0fd0-edc7-4b1a-a06a-8f794146edd0" path="/var/lib/kubelet/pods/056a0fd0-edc7-4b1a-a06a-8f794146edd0/volumes" Oct 30 18:18:36.870538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:36.858563 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9fb55b43-dcfe-4a5d-afba-f720ec95fbf4" path="/var/lib/kubelet/pods/9fb55b43-dcfe-4a5d-afba-f720ec95fbf4/volumes" Oct 30 18:18:37.114476 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.114442 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7"] Oct 30 18:18:37.120663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.120593 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7" Oct 30 18:18:37.135911 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.135888 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7"] Oct 30 18:18:37.141246 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.141204 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-a90958fa\"" Oct 30 18:18:37.141381 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.141273 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-a90958fa-dockercfg-hwrnb\"" Oct 30 18:18:37.174565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.174522 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n5f4d\" (UniqueName: \"kubernetes.io/projected/13bcd21f-ccd6-4d11-bb75-de03bc2683e6-kube-api-access-n5f4d\") pod \"payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7\" (UID: \"13bcd21f-ccd6-4d11-bb75-de03bc2683e6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7" Oct 30 18:18:37.174706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.174672 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13bcd21f-ccd6-4d11-bb75-de03bc2683e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7\" (UID: \"13bcd21f-ccd6-4d11-bb75-de03bc2683e6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7" Oct 30 18:18:37.286533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.280826 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n5f4d\" (UniqueName: \"kubernetes.io/projected/13bcd21f-ccd6-4d11-bb75-de03bc2683e6-kube-api-access-n5f4d\") pod \"payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7\" (UID: \"13bcd21f-ccd6-4d11-bb75-de03bc2683e6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7" Oct 30 18:18:37.286533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.280948 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13bcd21f-ccd6-4d11-bb75-de03bc2683e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7\" (UID: \"13bcd21f-ccd6-4d11-bb75-de03bc2683e6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7" Oct 30 18:18:37.286533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.281940 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13bcd21f-ccd6-4d11-bb75-de03bc2683e6-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7\" (UID: \"13bcd21f-ccd6-4d11-bb75-de03bc2683e6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7" Oct 30 18:18:37.345532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.335249 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n5f4d\" (UniqueName: \"kubernetes.io/projected/13bcd21f-ccd6-4d11-bb75-de03bc2683e6-kube-api-access-n5f4d\") pod \"payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7\" (UID: \"13bcd21f-ccd6-4d11-bb75-de03bc2683e6\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7" Oct 30 18:18:37.367697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.367660 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj"] Oct 30 18:18:37.398929 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.398851 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj" Oct 30 18:18:37.448537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.448135 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj"] Oct 30 18:18:37.448537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.448455 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7" Oct 30 18:18:37.641543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.633080 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6wgr\" (UniqueName: \"kubernetes.io/projected/70304573-b29c-45ef-933e-f974d4bac229-kube-api-access-g6wgr\") pod \"payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj\" (UID: \"70304573-b29c-45ef-933e-f974d4bac229\") " pod="openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj" Oct 30 18:18:37.641543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.633211 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70304573-b29c-45ef-933e-f974d4bac229-plugins-config-volume\") pod \"payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj\" (UID: \"70304573-b29c-45ef-933e-f974d4bac229\") " pod="openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj" Oct 30 18:18:37.743454 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.743422 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6wgr\" (UniqueName: \"kubernetes.io/projected/70304573-b29c-45ef-933e-f974d4bac229-kube-api-access-g6wgr\") pod \"payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj\" (UID: \"70304573-b29c-45ef-933e-f974d4bac229\") " pod="openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj" Oct 30 18:18:37.743639 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.743554 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70304573-b29c-45ef-933e-f974d4bac229-plugins-config-volume\") pod \"payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj\" (UID: \"70304573-b29c-45ef-933e-f974d4bac229\") " pod="openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj" Oct 30 18:18:37.744408 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.744387 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70304573-b29c-45ef-933e-f974d4bac229-plugins-config-volume\") pod \"payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj\" (UID: \"70304573-b29c-45ef-933e-f974d4bac229\") " pod="openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj" Oct 30 18:18:37.762067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.762043 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6wgr\" (UniqueName: \"kubernetes.io/projected/70304573-b29c-45ef-933e-f974d4bac229-kube-api-access-g6wgr\") pod \"payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj\" (UID: \"70304573-b29c-45ef-933e-f974d4bac229\") " pod="openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj" Oct 30 18:18:37.965751 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.965722 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7"] Oct 30 18:18:37.966629 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:18:37.966569 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13bcd21f_ccd6_4d11_bb75_de03bc2683e6.slice/crio-c2234b290e4e7adda65dfd6d0099118a1dc237e7ed66a016815a4f62bc942c64 WatchSource:0}: Error finding container c2234b290e4e7adda65dfd6d0099118a1dc237e7ed66a016815a4f62bc942c64: Status 404 returned error can't find the container with id c2234b290e4e7adda65dfd6d0099118a1dc237e7ed66a016815a4f62bc942c64 Oct 30 18:18:37.982116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:37.982080 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj" Oct 30 18:18:38.030083 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:38.030049 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg"] Oct 30 18:18:38.037040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:38.037013 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-20f778b3-dnxfg"] Oct 30 18:18:38.293326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:38.293298 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj"] Oct 30 18:18:38.293730 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:18:38.293704 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod70304573_b29c_45ef_933e_f974d4bac229.slice/crio-2d1c24c0f67dcca47fdfe719742651add5b7a57991d4fe0783b4d160610a8ab0 WatchSource:0}: Error finding container 2d1c24c0f67dcca47fdfe719742651add5b7a57991d4fe0783b4d160610a8ab0: Status 404 returned error can't find the container with id 2d1c24c0f67dcca47fdfe719742651add5b7a57991d4fe0783b4d160610a8ab0 Oct 30 18:18:38.758545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:38.754738 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj" event={"ID":"70304573-b29c-45ef-933e-f974d4bac229","Type":"ContainerStarted","Data":"41cb178b155bd3f77bcb0ad2ee393b95deb0d5b6317994fad0dd5bf0273d4cca"} Oct 30 18:18:38.758545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:38.754810 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj" event={"ID":"70304573-b29c-45ef-933e-f974d4bac229","Type":"ContainerStarted","Data":"2d1c24c0f67dcca47fdfe719742651add5b7a57991d4fe0783b4d160610a8ab0"} Oct 30 18:18:38.758545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:38.756094 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj" Oct 30 18:18:38.766539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:38.762244 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7" event={"ID":"13bcd21f-ccd6-4d11-bb75-de03bc2683e6","Type":"ContainerStarted","Data":"b5285393ae8c28349174417e7c41d6c46a7c3ccb74b2afba8bd3927d7abb8a36"} Oct 30 18:18:38.766539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:38.762315 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7" event={"ID":"13bcd21f-ccd6-4d11-bb75-de03bc2683e6","Type":"ContainerStarted","Data":"c2234b290e4e7adda65dfd6d0099118a1dc237e7ed66a016815a4f62bc942c64"} Oct 30 18:18:38.766539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:38.763231 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7" Oct 30 18:18:38.846804 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:38.846714 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc03de8a-fe28-4db9-9f5b-730beac01d95" path="/var/lib/kubelet/pods/fc03de8a-fe28-4db9-9f5b-730beac01d95/volumes" Oct 30 18:18:38.859533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:38.857155 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj" podStartSLOduration=1.857134182 podStartE2EDuration="1.857134182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:38.801098242 +0000 UTC m=+1018.117518568" watchObservedRunningTime="2026-10-30 18:18:38.857134182 +0000 UTC m=+1018.173554504" Oct 30 18:18:38.927166 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:38.927107 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7" podStartSLOduration=1.927092408 podStartE2EDuration="1.927092408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:38.841611414 +0000 UTC m=+1018.158031736" watchObservedRunningTime="2026-10-30 18:18:38.927092408 +0000 UTC m=+1018.243512764" Oct 30 18:18:40.261573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:40.237092 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2"] Oct 30 18:18:40.361534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:40.335860 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2" Oct 30 18:18:40.388537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:40.378252 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:18:40.388537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:40.378301 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:18:40.490543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:40.488561 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2"] Oct 30 18:18:40.494538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:40.491446 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75c2eac-859e-4b3e-8b85-f939190e70fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-535b743b-l62q2\" (UID: \"c75c2eac-859e-4b3e-8b85-f939190e70fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2" Oct 30 18:18:40.494538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:40.491526 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q648b\" (UniqueName: \"kubernetes.io/projected/c75c2eac-859e-4b3e-8b85-f939190e70fc-kube-api-access-q648b\") pod \"maas-api-revoke-keys-e2e-migrate-535b743b-l62q2\" (UID: \"c75c2eac-859e-4b3e-8b85-f939190e70fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2" Oct 30 18:18:40.494538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:40.491608 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75c2eac-859e-4b3e-8b85-f939190e70fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-535b743b-l62q2\" (UID: \"c75c2eac-859e-4b3e-8b85-f939190e70fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2" Oct 30 18:18:40.634585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:40.631333 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75c2eac-859e-4b3e-8b85-f939190e70fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-535b743b-l62q2\" (UID: \"c75c2eac-859e-4b3e-8b85-f939190e70fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2" Oct 30 18:18:40.634585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:40.631397 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q648b\" (UniqueName: \"kubernetes.io/projected/c75c2eac-859e-4b3e-8b85-f939190e70fc-kube-api-access-q648b\") pod \"maas-api-revoke-keys-e2e-migrate-535b743b-l62q2\" (UID: \"c75c2eac-859e-4b3e-8b85-f939190e70fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2" Oct 30 18:18:40.634585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:40.631464 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75c2eac-859e-4b3e-8b85-f939190e70fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-535b743b-l62q2\" (UID: \"c75c2eac-859e-4b3e-8b85-f939190e70fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2" Oct 30 18:18:40.711321 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:40.711286 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75c2eac-859e-4b3e-8b85-f939190e70fc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-535b743b-l62q2\" (UID: \"c75c2eac-859e-4b3e-8b85-f939190e70fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2" Oct 30 18:18:40.814473 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:40.814421 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:40.919331 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:40.917186 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q648b\" (UniqueName: \"kubernetes.io/projected/c75c2eac-859e-4b3e-8b85-f939190e70fc-kube-api-access-q648b\") pod \"maas-api-revoke-keys-e2e-migrate-535b743b-l62q2\" (UID: \"c75c2eac-859e-4b3e-8b85-f939190e70fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2" Oct 30 18:18:40.919331 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:40.917322 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:18:40.976707 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:40.976664 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.002582 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:40.982775 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75c2eac-859e-4b3e-8b85-f939190e70fc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-535b743b-l62q2\" (UID: \"c75c2eac-859e-4b3e-8b85-f939190e70fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2" Oct 30 18:18:41.071129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:41.043848 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.208427 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:41.208338 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:18:41.208682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:41.208654 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2" Oct 30 18:18:41.347453 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:41.345731 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.357604 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:41.348681 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.436651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:41.436607 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.461726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:41.461637 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.512499 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:41.512382 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:18:41.640643 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:41.632380 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.833537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:41.825664 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:18:41.833537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:41.826258 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:18:41.942833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:41.942617 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:41.964535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:41.943288 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:18:42.049535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:42.046702 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.049535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:42.047300 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:18:42.311816 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:42.311748 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.347486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:42.347443 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.369714 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:42.369672 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:18:42.389551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:42.376113 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.408827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:42.408801 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:18:42.475362 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:42.454912 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.476676 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:42.476632 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:18:42.477395 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:42.477046 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:18:42.477762 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:42.477739 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:18:42.545750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:42.545692 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.574584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:42.546417 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:18:42.574584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:42.547048 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:18:42.574584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:42.570125 5232 scope.go:122] "RemoveContainer" containerID="7a181cb1ab898eeada4c2f7a2fc5d3024c9fcddffa8b4ae423965eff540b2734" Oct 30 18:18:42.782861 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:42.782810 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:42.980996 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:42.980719 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.047827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:43.047748 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.229622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:43.229569 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 18:18:43.633599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:43.509661 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:18:43.633599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:43.509770 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.633599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:43.511599 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.633599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:43.599644 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:18:43.633599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:43.604048 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:18:43.699603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:43.673042 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-embedding-simulated-kserve-67bf594cd6-7k4cr" podUID="1b2b3e72-d4ac-4407-97ae-421c5ad091a1" containerName="main" probeResult="failure" output="Get \"https://10.133.0.25:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:43.758736 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:43.758696 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:18:43.767909 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:43.767882 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:18:44.574572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:43.783702 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.574572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:43.810690 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:18:44.635538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:44.018387 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.661598 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:44.088180 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:18:44.736368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:44.701209 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.848602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:44.612994 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:18:44.848602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:44.612864 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.848602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:44.810625 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 18:18:44.871838 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:44.871748 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:44.901610 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:44.875959 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:18:44.902439 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:44.902398 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:18:45.082687 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:45.082640 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": context deadline exceeded" Oct 30 18:18:45.550239 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:18:45.550150 5232 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.014s" Oct 30 18:18:45.577655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:45.577611 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:18:45.578984 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:45.578950 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:18:45.717539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:45.681352 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:18:45.732640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:45.732595 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:18:45.817978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:45.816666 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.837665 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:45.837620 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.844542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:45.838398 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.875714 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:45.875675 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:45.901536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:45.893048 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:18:45.902495 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:45.902471 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:18:45.965541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:45.940844 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:46.062776 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:45.954838 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:46.075595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:46.030644 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:46.662711 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:46.629626 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:18:46.725007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:46.724953 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:18:46.773854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:46.772585 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:18:46.818701 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:46.779250 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:47.014614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:46.823544 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:18:47.058823 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:46.989861 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:18:47.058823 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:46.990437 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:18:47.094536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:47.065988 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:47.261675 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:47.093643 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:18:47.261675 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:46.989988 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:47.471753 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:47.450112 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 18:18:47.581538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:47.509842 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:47.584591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:47.583980 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded" Oct 30 18:18:47.655928 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:47.655703 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:47.742594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:47.740437 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:47.790168 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:47.790122 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:18:47.845676 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:18:47.845258 5232 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.048s" Oct 30 18:18:47.846000 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:47.845981 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:18:47.846163 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:47.846140 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2" event={"ID":"c75c2eac-859e-4b3e-8b85-f939190e70fc","Type":"ContainerStarted","Data":"2a57e633acc1d7f311a4439f6cdd4672b25514751ef0867a931e7f06938a0d5e"} Oct 30 18:18:47.891547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:47.882741 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2"] Oct 30 18:18:48.001973 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:48.001812 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:18:48.018654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:48.018598 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:48.045829 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:48.044665 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:48.045829 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:48.045374 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:18:48.150642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:48.149304 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:48.359005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:48.240667 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:18:48.359005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:48.241229 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:48.359005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:48.241418 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:18:48.456775 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:48.456724 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:48.524736 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:48.524695 5232 patch_prober.go:29] interesting pod/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:48.525012 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:48.524979 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" podUID="03bc2ff1-ddb6-41cf-8238-4df762a46c0e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.91:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:48.666740 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:48.658624 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:48.721032 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:48.720383 5232 generic.go:356] "Generic (PLEG): container finished" podID="c75c2eac-859e-4b3e-8b85-f939190e70fc" containerID="b32b9216c48eb085ea7fee9803dd6a09566f0ae3962e99c3a4d5c59322966421" exitCode=0 Oct 30 18:18:48.721032 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:48.720565 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2" event={"ID":"c75c2eac-859e-4b3e-8b85-f939190e70fc","Type":"ContainerDied","Data":"b32b9216c48eb085ea7fee9803dd6a09566f0ae3962e99c3a4d5c59322966421"} Oct 30 18:18:48.721619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:48.721393 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": context deadline exceeded" Oct 30 18:18:48.752635 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:48.752589 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:48.868538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:48.866614 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.010171 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:49.010128 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.069471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:49.069419 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:18:49.070559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:49.069732 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:18:49.127535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:49.107589 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9"] Oct 30 18:18:49.148744 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:49.148676 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.149527 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:49.149477 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.163547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:49.161980 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-704e738d-pssq9"] Oct 30 18:18:49.242646 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:49.242604 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.243318 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:49.243282 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:18:49.313790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:49.313693 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.349117 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:49.349078 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:18:49.409670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:49.409623 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.681742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:49.681649 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.745561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:49.718552 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:18:49.757668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:49.754717 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:49.961679 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:49.961578 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:50.044754 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.044709 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:50.046442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.046410 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:50.077531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.073763 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:50.077531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.074130 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:18:50.109003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.107291 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:50.109003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.107483 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:18:50.109003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.108733 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:18:50.162535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.161854 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:18:50.188543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.187621 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:50.188543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.187987 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:18:50.196910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.196619 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:18:50.204314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.204290 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz"] Oct 30 18:18:50.207116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.207095 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:18:50.247343 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.247152 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c06fd47-8nwnz"] Oct 30 18:18:50.253782 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.253752 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:18:50.255592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.255571 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:18:50.258045 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.258026 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:18:50.469570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.469540 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2" Oct 30 18:18:50.602936 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.602858 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c75c2eac-859e-4b3e-8b85-f939190e70fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75c2eac-859e-4b3e-8b85-f939190e70fc-maas-api-xks-ca\") pod \"c75c2eac-859e-4b3e-8b85-f939190e70fc\" (UID: \"c75c2eac-859e-4b3e-8b85-f939190e70fc\") " Oct 30 18:18:50.603075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.603010 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c75c2eac-859e-4b3e-8b85-f939190e70fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75c2eac-859e-4b3e-8b85-f939190e70fc-maas-api-openshift-ca\") pod \"c75c2eac-859e-4b3e-8b85-f939190e70fc\" (UID: \"c75c2eac-859e-4b3e-8b85-f939190e70fc\") " Oct 30 18:18:50.603122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.603067 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c75c2eac-859e-4b3e-8b85-f939190e70fc-kube-api-access-q648b\" (UniqueName: \"kubernetes.io/projected/c75c2eac-859e-4b3e-8b85-f939190e70fc-kube-api-access-q648b\") pod \"c75c2eac-859e-4b3e-8b85-f939190e70fc\" (UID: \"c75c2eac-859e-4b3e-8b85-f939190e70fc\") " Oct 30 18:18:50.604397 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.604371 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c75c2eac-859e-4b3e-8b85-f939190e70fc-maas-api-openshift-ca" pod "c75c2eac-859e-4b3e-8b85-f939190e70fc" (UID: "c75c2eac-859e-4b3e-8b85-f939190e70fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:18:50.605491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.605471 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c75c2eac-859e-4b3e-8b85-f939190e70fc-maas-api-xks-ca" pod "c75c2eac-859e-4b3e-8b85-f939190e70fc" (UID: "c75c2eac-859e-4b3e-8b85-f939190e70fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:18:50.606830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.606808 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c75c2eac-859e-4b3e-8b85-f939190e70fc-kube-api-access-q648b" pod "c75c2eac-859e-4b3e-8b85-f939190e70fc" (UID: "c75c2eac-859e-4b3e-8b85-f939190e70fc"). InnerVolumeSpecName "kube-api-access-q648b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:18:50.705244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.705213 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c75c2eac-859e-4b3e-8b85-f939190e70fc-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:18:50.705244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.705241 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q648b\" (UniqueName: \"kubernetes.io/projected/c75c2eac-859e-4b3e-8b85-f939190e70fc-kube-api-access-q648b\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:18:50.705244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.705252 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c75c2eac-859e-4b3e-8b85-f939190e70fc-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:18:50.940314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.940219 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fa09905-d3a9-42b9-9afe-3745e1bbaf71" path="/var/lib/kubelet/pods/2fa09905-d3a9-42b9-9afe-3745e1bbaf71/volumes" Oct 30 18:18:50.943410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.943361 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17" path="/var/lib/kubelet/pods/a29a2b2a-cd25-4f9b-bfc0-bca1482cdb17/volumes" Oct 30 18:18:50.958892 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.958871 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2" Oct 30 18:18:50.959072 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.959040 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2" event={"ID":"c75c2eac-859e-4b3e-8b85-f939190e70fc","Type":"ContainerDied","Data":"2a57e633acc1d7f311a4439f6cdd4672b25514751ef0867a931e7f06938a0d5e"} Oct 30 18:18:50.959350 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:50.959319 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a57e633acc1d7f311a4439f6cdd4672b25514751ef0867a931e7f06938a0d5e" Oct 30 18:18:51.601829 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:51.601803 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj" Oct 30 18:18:51.637773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:51.637742 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7" Oct 30 18:18:54.156535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:54.150200 5232 patch_prober.go:29] interesting pod/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:18:54.156535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:54.150269 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" podUID="3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:55.028973 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:55.028924 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb"] Oct 30 18:18:55.043606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:55.043558 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-559a064f-f47wb"] Oct 30 18:18:56.032278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.032246 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2"] Oct 30 18:18:56.044995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.044965 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07c65b1a-tppn2"] Oct 30 18:18:56.248563 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.248522 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf"] Oct 30 18:18:56.249590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.249568 5232 cpu_manager.go:414] "containerMap: removing container" podUID="c75c2eac-859e-4b3e-8b85-f939190e70fc" containerName="revoke-keys" Oct 30 18:18:56.249654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.249592 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c75c2eac-859e-4b3e-8b85-f939190e70fc" containerName="revoke-keys" Oct 30 18:18:56.249741 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.249732 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="c75c2eac-859e-4b3e-8b85-f939190e70fc" containerName="revoke-keys" Oct 30 18:18:56.252670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.252655 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.266020 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.265980 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d"] Oct 30 18:18:56.276626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.276593 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-7cb9c50e-openshift-default-dockercfg-rvfbm\"" Oct 30 18:18:56.276916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.276893 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.327900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.327824 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf"] Oct 30 18:18:56.372009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.371610 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d"] Oct 30 18:18:56.424803 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.424757 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-credential-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.424987 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.424826 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-envoy\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.424987 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.424945 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rcgps\" (UniqueName: \"kubernetes.io/projected/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-kube-api-access-rcgps\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.425091 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.425001 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-token\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.425091 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.425077 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f187b761-f4e5-4126-a03d-5f4418db9cf0-istiod-ca-cert\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.425166 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.425114 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-workload-certs\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.425209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.425173 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sdsz\" (UniqueName: \"kubernetes.io/projected/f187b761-f4e5-4126-a03d-5f4418db9cf0-kube-api-access-4sdsz\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.425265 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.425215 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-data\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.425265 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.425258 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-data\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.425347 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.425279 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-workload-certs\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.425382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.425344 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-podinfo\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.425416 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.425394 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-podinfo\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.425494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.425465 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-workload-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.425601 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.425552 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-token\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.425601 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.425579 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istiod-ca-cert\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.425730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.425703 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-credential-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.425825 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.425806 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-envoy\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.425898 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.425881 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-workload-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.541461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.528714 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-envoy\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.541461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.528825 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-workload-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.541461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.528945 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-credential-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.541461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.529010 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-envoy\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.541461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.529078 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rcgps\" (UniqueName: \"kubernetes.io/projected/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-kube-api-access-rcgps\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.541461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.529116 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-token\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.541461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.529173 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f187b761-f4e5-4126-a03d-5f4418db9cf0-istiod-ca-cert\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.541461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.529212 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-workload-certs\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.541461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.529246 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sdsz\" (UniqueName: \"kubernetes.io/projected/f187b761-f4e5-4126-a03d-5f4418db9cf0-kube-api-access-4sdsz\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.541461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.529300 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-data\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.541461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.529375 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-data\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.541461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.529414 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-workload-certs\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.541461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.529457 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-podinfo\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.541461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.530459 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-credential-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.561533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.558367 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-podinfo\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.561533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.558425 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-workload-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.561533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.558452 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-token\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.561533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.558479 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istiod-ca-cert\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.561533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.558829 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-data\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.561533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.559687 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-workload-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.561533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.560465 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f187b761-f4e5-4126-a03d-5f4418db9cf0-istiod-ca-cert\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.561533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.560730 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-workload-certs\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.561533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.560969 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-data\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.561533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.561184 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-workload-certs\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.561533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.561407 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-envoy\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.581304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.581051 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-workload-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.596532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.590921 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-credential-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.596532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.591464 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-credential-socket\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.606535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.604908 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istiod-ca-cert\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.626533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.623620 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-token\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.643532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.631168 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-envoy\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.643532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.631860 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-token\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.643532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.632368 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sdsz\" (UniqueName: \"kubernetes.io/projected/f187b761-f4e5-4126-a03d-5f4418db9cf0-kube-api-access-4sdsz\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.643532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.640051 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-podinfo\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.643532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.643242 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rcgps\" (UniqueName: \"kubernetes.io/projected/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-kube-api-access-rcgps\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.656533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.649162 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-podinfo\") pod \"e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:56.682565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.675644 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:56.815984 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.815177 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4790a52-fcfc-4d32-becc-61ff1eb5ddd3" path="/var/lib/kubelet/pods/a4790a52-fcfc-4d32-becc-61ff1eb5ddd3/volumes" Oct 30 18:18:56.816965 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.816941 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c44655c7-466e-419a-a2b4-04d6f11df1d6" path="/var/lib/kubelet/pods/c44655c7-466e-419a-a2b4-04d6f11df1d6/volumes" Oct 30 18:18:56.883758 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:56.883673 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:57.046759 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.046721 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d"] Oct 30 18:18:57.053072 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:18:57.053037 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:18:57.053611 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:18:57.053573 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf187b761_f4e5_4126_a03d_5f4418db9cf0.slice/crio-a26734e2c09643b3d0d08bf13f218836c01b99772ff53c8e77328e3e8ff5d0d1 WatchSource:0}: Error finding container a26734e2c09643b3d0d08bf13f218836c01b99772ff53c8e77328e3e8ff5d0d1: Status 404 returned error can't find the container with id a26734e2c09643b3d0d08bf13f218836c01b99772ff53c8e77328e3e8ff5d0d1 Oct 30 18:18:57.056402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.056366 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:18:57.056541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.056454 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:18:57.056541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.056501 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:18:57.218445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.218408 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:18:57.218636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.218523 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:18:57.218636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.218555 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf"] Oct 30 18:18:57.218636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.218559 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:18:57.304791 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.304743 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr"] Oct 30 18:18:57.308887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.308858 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr" Oct 30 18:18:57.323723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.323691 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:18:57.323896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.323812 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:18:57.326259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.326228 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr"] Oct 30 18:18:57.411611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.411462 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27jjv\" (UniqueName: \"kubernetes.io/projected/44c9b672-70cf-40ce-aa40-f06c8462273e-kube-api-access-27jjv\") pod \"maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr\" (UID: \"44c9b672-70cf-40ce-aa40-f06c8462273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr" Oct 30 18:18:57.411772 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.411755 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c9b672-70cf-40ce-aa40-f06c8462273e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr\" (UID: \"44c9b672-70cf-40ce-aa40-f06c8462273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr" Oct 30 18:18:57.411820 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.411795 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c9b672-70cf-40ce-aa40-f06c8462273e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr\" (UID: \"44c9b672-70cf-40ce-aa40-f06c8462273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr" Oct 30 18:18:57.513809 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.513772 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27jjv\" (UniqueName: \"kubernetes.io/projected/44c9b672-70cf-40ce-aa40-f06c8462273e-kube-api-access-27jjv\") pod \"maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr\" (UID: \"44c9b672-70cf-40ce-aa40-f06c8462273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr" Oct 30 18:18:57.514606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.514241 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c9b672-70cf-40ce-aa40-f06c8462273e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr\" (UID: \"44c9b672-70cf-40ce-aa40-f06c8462273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr" Oct 30 18:18:57.514606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.514336 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c9b672-70cf-40ce-aa40-f06c8462273e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr\" (UID: \"44c9b672-70cf-40ce-aa40-f06c8462273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr" Oct 30 18:18:57.515633 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.515606 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c9b672-70cf-40ce-aa40-f06c8462273e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr\" (UID: \"44c9b672-70cf-40ce-aa40-f06c8462273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr" Oct 30 18:18:57.517233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.517210 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c9b672-70cf-40ce-aa40-f06c8462273e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr\" (UID: \"44c9b672-70cf-40ce-aa40-f06c8462273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr" Oct 30 18:18:57.536838 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.536799 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27jjv\" (UniqueName: \"kubernetes.io/projected/44c9b672-70cf-40ce-aa40-f06c8462273e-kube-api-access-27jjv\") pod \"maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr\" (UID: \"44c9b672-70cf-40ce-aa40-f06c8462273e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr" Oct 30 18:18:57.678040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.677948 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr" Oct 30 18:18:57.997559 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:18:57.991243 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44c9b672_70cf_40ce_aa40_f06c8462273e.slice/crio-e957d12c04bbc42ffd53d43eec4286aa0aa56194edcea3392a7836f3a8394ef6 WatchSource:0}: Error finding container e957d12c04bbc42ffd53d43eec4286aa0aa56194edcea3392a7836f3a8394ef6: Status 404 returned error can't find the container with id e957d12c04bbc42ffd53d43eec4286aa0aa56194edcea3392a7836f3a8394ef6 Oct 30 18:18:57.997559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:57.992672 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr"] Oct 30 18:18:58.074563 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:58.073833 5232 generic.go:356] "Generic (PLEG): container finished" podID="3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" containerID="bc9065ec30dd487b69207a9afcb3e5f34eabe7897fc972d52595d651e7a70e7d" exitCode=0 Oct 30 18:18:58.074563 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:58.074082 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" event={"ID":"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984","Type":"ContainerDied","Data":"bc9065ec30dd487b69207a9afcb3e5f34eabe7897fc972d52595d651e7a70e7d"} Oct 30 18:18:58.075133 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:58.074730 5232 scope.go:122] "RemoveContainer" containerID="bc9065ec30dd487b69207a9afcb3e5f34eabe7897fc972d52595d651e7a70e7d" Oct 30 18:18:58.083322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:58.078335 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:18:58.083322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:58.079234 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:18:58.083322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:58.080400 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:18:58.083322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:58.081680 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" event={"ID":"f187b761-f4e5-4126-a03d-5f4418db9cf0","Type":"ContainerStarted","Data":"4c18324f59f7a39fc8608597903af09c1981106b77afb336ebdd50810ba9c054"} Oct 30 18:18:58.083322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:58.081740 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" event={"ID":"f187b761-f4e5-4126-a03d-5f4418db9cf0","Type":"ContainerStarted","Data":"a26734e2c09643b3d0d08bf13f218836c01b99772ff53c8e77328e3e8ff5d0d1"} Oct 30 18:18:58.087538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:58.084847 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr" event={"ID":"44c9b672-70cf-40ce-aa40-f06c8462273e","Type":"ContainerStarted","Data":"e957d12c04bbc42ffd53d43eec4286aa0aa56194edcea3392a7836f3a8394ef6"} Oct 30 18:18:58.087538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:58.087113 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" event={"ID":"fe9253e4-0ac7-47ba-85d8-b55a75b5334b","Type":"ContainerStarted","Data":"a18a817d9325883ec44f6b37b9d69951e3e0718dfb0ed51a20ddab2d78578881"} Oct 30 18:18:58.087538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:58.087165 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" event={"ID":"fe9253e4-0ac7-47ba-85d8-b55a75b5334b","Type":"ContainerStarted","Data":"2d1f7e593a64164bb55ee32a2fd69486f0476ce833b62d8fbf86066671297459"} Oct 30 18:18:58.298846 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:58.298786 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" podStartSLOduration=2.298769847 podStartE2EDuration="2.298769847s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:58.298041288 +0000 UTC m=+1037.614461599" watchObservedRunningTime="2026-10-30 18:18:58.298769847 +0000 UTC m=+1037.615190159" Oct 30 18:18:58.304276 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:58.304232 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" podStartSLOduration=2.304202467 podStartE2EDuration="2.304202467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:58.209495257 +0000 UTC m=+1037.525915594" watchObservedRunningTime="2026-10-30 18:18:58.304202467 +0000 UTC m=+1037.620622783" Oct 30 18:18:58.678597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:58.677892 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:18:58.752551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:58.737004 5232 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 18:18:58.752551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:58.737070 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" podUID="f187b761-f4e5-4126-a03d-5f4418db9cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 18:18:59.046416 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:58.909876 5232 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 18:18:59.046416 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:58.909943 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" podUID="fe9253e4-0ac7-47ba-85d8-b55a75b5334b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 18:18:59.281233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:59.281165 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr" event={"ID":"44c9b672-70cf-40ce-aa40-f06c8462273e","Type":"ContainerStarted","Data":"04b3ad726c57bc91c23613c8acad7eb23abb5393d67149e878006af67df5d815"} Oct 30 18:18:59.283122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:59.283089 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:18:59.286738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:59.286690 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" event={"ID":"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984","Type":"ContainerStarted","Data":"6562083aeef1ca9679b8d96bd3d803973bc6c476f775b6c8a4f7ca9d4ad1286a"} Oct 30 18:18:59.367661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:59.363899 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:18:59.386527 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:59.384029 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr" podStartSLOduration=2.384007572 podStartE2EDuration="2.384007572s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:18:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:18:59.338845205 +0000 UTC m=+1038.655265530" watchObservedRunningTime="2026-10-30 18:18:59.384007572 +0000 UTC m=+1038.700427891" Oct 30 18:18:59.676786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:59.676702 5232 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 18:18:59.676786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:59.676750 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" podUID="f187b761-f4e5-4126-a03d-5f4418db9cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 18:18:59.885530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:59.884793 5232 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 18:18:59.885530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:18:59.884887 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" podUID="fe9253e4-0ac7-47ba-85d8-b55a75b5334b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 18:19:00.252540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:00.250920 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:19:00.278538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:00.254701 5232 patch_prober.go:29] interesting pod/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:19:00.278538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:00.254778 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" podUID="3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 18:19:00.345056 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:00.345011 5232 generic.go:356] "Generic (PLEG): container finished" podID="44c9b672-70cf-40ce-aa40-f06c8462273e" containerID="04b3ad726c57bc91c23613c8acad7eb23abb5393d67149e878006af67df5d815" exitCode=0 Oct 30 18:19:00.362406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:00.358565 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr" event={"ID":"44c9b672-70cf-40ce-aa40-f06c8462273e","Type":"ContainerDied","Data":"04b3ad726c57bc91c23613c8acad7eb23abb5393d67149e878006af67df5d815"} Oct 30 18:19:00.687594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:00.683884 5232 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 18:19:00.708534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:00.687949 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" podUID="f187b761-f4e5-4126-a03d-5f4418db9cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 18:19:00.887737 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:00.887704 5232 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 18:19:00.888053 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:00.888008 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" podUID="fe9253e4-0ac7-47ba-85d8-b55a75b5334b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 18:19:01.236097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:01.236061 5232 patch_prober.go:29] interesting pod/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:19:01.236421 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:01.236371 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" podUID="3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 18:19:01.705581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:01.688741 5232 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 18:19:01.705581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:01.688799 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" podUID="f187b761-f4e5-4126-a03d-5f4418db9cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 18:19:01.902251 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:01.894543 5232 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 18:19:01.902251 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:01.894614 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" podUID="fe9253e4-0ac7-47ba-85d8-b55a75b5334b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 18:19:02.107828 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:02.107694 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp"] Oct 30 18:19:02.345907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:02.345870 5232 patch_prober.go:29] interesting pod/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:19:02.346186 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:02.346154 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" podUID="3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 18:19:02.665471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:02.639434 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-355285a5-xhcrp"] Oct 30 18:19:02.715555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:02.709560 5232 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 18:19:02.715555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:02.709658 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" podUID="f187b761-f4e5-4126-a03d-5f4418db9cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 18:19:02.871562 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:02.870201 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:19:02.871562 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:02.870587 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:19:03.127342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.066554 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:19:03.127342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.077922 5232 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 18:19:03.127342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.077977 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" podUID="fe9253e4-0ac7-47ba-85d8-b55a75b5334b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 18:19:03.127342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.101153 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0a474cc-27db-44de-a436-e9d490d78ec1" path="/var/lib/kubelet/pods/a0a474cc-27db-44de-a436-e9d490d78ec1/volumes" Oct 30 18:19:03.234687 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.231081 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr" Oct 30 18:19:03.356536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.350704 5232 patch_prober.go:29] interesting pod/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:19:03.356536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.350785 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" podUID="3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 18:19:03.496572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.489437 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/44c9b672-70cf-40ce-aa40-f06c8462273e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c9b672-70cf-40ce-aa40-f06c8462273e-maas-api-xks-ca\") pod \"44c9b672-70cf-40ce-aa40-f06c8462273e\" (UID: \"44c9b672-70cf-40ce-aa40-f06c8462273e\") " Oct 30 18:19:03.496572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.489646 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44c9b672-70cf-40ce-aa40-f06c8462273e-kube-api-access-27jjv\" (UniqueName: \"kubernetes.io/projected/44c9b672-70cf-40ce-aa40-f06c8462273e-kube-api-access-27jjv\") pod \"44c9b672-70cf-40ce-aa40-f06c8462273e\" (UID: \"44c9b672-70cf-40ce-aa40-f06c8462273e\") " Oct 30 18:19:03.496572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.489714 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44c9b672-70cf-40ce-aa40-f06c8462273e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c9b672-70cf-40ce-aa40-f06c8462273e-maas-api-openshift-ca\") pod \"44c9b672-70cf-40ce-aa40-f06c8462273e\" (UID: \"44c9b672-70cf-40ce-aa40-f06c8462273e\") " Oct 30 18:19:03.496572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.495717 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44c9b672-70cf-40ce-aa40-f06c8462273e-maas-api-openshift-ca" pod "44c9b672-70cf-40ce-aa40-f06c8462273e" (UID: "44c9b672-70cf-40ce-aa40-f06c8462273e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:03.517953 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.517903 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:19:03.574532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.553543 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/44c9b672-70cf-40ce-aa40-f06c8462273e-maas-api-xks-ca" pod "44c9b672-70cf-40ce-aa40-f06c8462273e" (UID: "44c9b672-70cf-40ce-aa40-f06c8462273e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:03.615834 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.615050 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms"] Oct 30 18:19:03.627650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.627619 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/44c9b672-70cf-40ce-aa40-f06c8462273e-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:03.627887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.627869 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/44c9b672-70cf-40ce-aa40-f06c8462273e-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:03.709387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.706711 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44c9b672-70cf-40ce-aa40-f06c8462273e-kube-api-access-27jjv" pod "44c9b672-70cf-40ce-aa40-f06c8462273e" (UID: "44c9b672-70cf-40ce-aa40-f06c8462273e"). InnerVolumeSpecName "kube-api-access-27jjv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:03.797659 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.791581 5232 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 18:19:03.797659 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.791672 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" podUID="f187b761-f4e5-4126-a03d-5f4418db9cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 18:19:03.797659 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.793587 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:19:03.944533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.907932 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27jjv\" (UniqueName: \"kubernetes.io/projected/44c9b672-70cf-40ce-aa40-f06c8462273e-kube-api-access-27jjv\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:03.984535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:03.984209 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a8b7933b-7qcms"] Oct 30 18:19:04.015534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.010133 5232 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 18:19:04.015534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.010209 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" podUID="fe9253e4-0ac7-47ba-85d8-b55a75b5334b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 18:19:04.079283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.079186 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr" event={"ID":"44c9b672-70cf-40ce-aa40-f06c8462273e","Type":"ContainerDied","Data":"e957d12c04bbc42ffd53d43eec4286aa0aa56194edcea3392a7836f3a8394ef6"} Oct 30 18:19:04.079639 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.079614 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e957d12c04bbc42ffd53d43eec4286aa0aa56194edcea3392a7836f3a8394ef6" Oct 30 18:19:04.079888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.079873 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr" Oct 30 18:19:04.109024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.108988 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq"] Oct 30 18:19:04.114021 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.110153 5232 cpu_manager.go:414] "containerMap: removing container" podUID="44c9b672-70cf-40ce-aa40-f06c8462273e" containerName="revoke-keys" Oct 30 18:19:04.114021 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.110179 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44c9b672-70cf-40ce-aa40-f06c8462273e" containerName="revoke-keys" Oct 30 18:19:04.114021 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.110413 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="44c9b672-70cf-40ce-aa40-f06c8462273e" containerName="revoke-keys" Oct 30 18:19:04.119412 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.119386 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq" Oct 30 18:19:04.137998 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.137797 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:19:04.138844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.138599 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:19:04.201660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.201615 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq"] Oct 30 18:19:04.247534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.246927 5232 patch_prober.go:29] interesting pod/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:19:04.247534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.246988 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" podUID="3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 18:19:04.270536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.249987 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc5sp\" (UniqueName: \"kubernetes.io/projected/17778834-4065-463a-b5ba-b58d1d91f0d5-kube-api-access-bc5sp\") pod \"maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq\" (UID: \"17778834-4065-463a-b5ba-b58d1d91f0d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq" Oct 30 18:19:04.270536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.250044 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17778834-4065-463a-b5ba-b58d1d91f0d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq\" (UID: \"17778834-4065-463a-b5ba-b58d1d91f0d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq" Oct 30 18:19:04.270536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.250196 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17778834-4065-463a-b5ba-b58d1d91f0d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq\" (UID: \"17778834-4065-463a-b5ba-b58d1d91f0d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq" Oct 30 18:19:04.376590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.375951 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc5sp\" (UniqueName: \"kubernetes.io/projected/17778834-4065-463a-b5ba-b58d1d91f0d5-kube-api-access-bc5sp\") pod \"maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq\" (UID: \"17778834-4065-463a-b5ba-b58d1d91f0d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq" Oct 30 18:19:04.376590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.376022 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17778834-4065-463a-b5ba-b58d1d91f0d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq\" (UID: \"17778834-4065-463a-b5ba-b58d1d91f0d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq" Oct 30 18:19:04.376590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.376197 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17778834-4065-463a-b5ba-b58d1d91f0d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq\" (UID: \"17778834-4065-463a-b5ba-b58d1d91f0d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq" Oct 30 18:19:04.395533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.393320 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17778834-4065-463a-b5ba-b58d1d91f0d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq\" (UID: \"17778834-4065-463a-b5ba-b58d1d91f0d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq" Oct 30 18:19:04.398535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.397680 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17778834-4065-463a-b5ba-b58d1d91f0d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq\" (UID: \"17778834-4065-463a-b5ba-b58d1d91f0d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq" Oct 30 18:19:04.403975 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.403931 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc5sp\" (UniqueName: \"kubernetes.io/projected/17778834-4065-463a-b5ba-b58d1d91f0d5-kube-api-access-bc5sp\") pod \"maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq\" (UID: \"17778834-4065-463a-b5ba-b58d1d91f0d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq" Oct 30 18:19:04.502496 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.502468 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq" Oct 30 18:19:04.696580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.688917 5232 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" start-of-body= Oct 30 18:19:04.696580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.688982 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" podUID="f187b761-f4e5-4126-a03d-5f4418db9cf0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.139:15021/healthz/ready\": dial tcp 10.133.0.139:15021: connect: connection refused" Oct 30 18:19:04.904590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.901986 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7be05604-c569-49a4-9538-238d1728e104" path="/var/lib/kubelet/pods/7be05604-c569-49a4-9538-238d1728e104/volumes" Oct 30 18:19:04.904590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.902340 5232 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" start-of-body= Oct 30 18:19:04.904590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:04.902394 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" podUID="fe9253e4-0ac7-47ba-85d8-b55a75b5334b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.138:15021/healthz/ready\": dial tcp 10.133.0.138:15021: connect: connection refused" Oct 30 18:19:05.256536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:05.235893 5232 patch_prober.go:29] interesting pod/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:19:05.256536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:05.235954 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" podUID="3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 18:19:05.559491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:05.559462 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq"] Oct 30 18:19:05.683557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:05.681851 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:19:05.683557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:05.682824 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:19:05.684764 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:05.684740 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:19:05.933655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:05.911475 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:19:05.933655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:05.912688 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:19:05.933655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:05.914759 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:19:06.236606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:06.236120 5232 patch_prober.go:29] interesting pod/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" start-of-body= Oct 30 18:19:06.236606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:06.236180 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" podUID="3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.126:15021/healthz/ready\": dial tcp 10.133.0.126:15021: connect: connection refused" Oct 30 18:19:06.246471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:06.246440 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq" event={"ID":"17778834-4065-463a-b5ba-b58d1d91f0d5","Type":"ContainerStarted","Data":"c17daf60b29c766d686e09128df2187bd33b5bfabd572fe4c08b967e9298772a"} Oct 30 18:19:06.246661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:06.246560 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq" event={"ID":"17778834-4065-463a-b5ba-b58d1d91f0d5","Type":"ContainerStarted","Data":"383c089d3e04a9be6767afe08f1eb663560d09c4c9303904f93ae95744928b51"} Oct 30 18:19:06.284944 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:06.284892 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq" podStartSLOduration=3.284879168 podStartE2EDuration="3.284879168s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:06.284499655 +0000 UTC m=+1045.600919977" watchObservedRunningTime="2026-10-30 18:19:06.284879168 +0000 UTC m=+1045.601299489" Oct 30 18:19:07.240095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:07.240067 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:19:07.240639 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:07.240623 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:19:07.241379 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:07.241359 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:19:07.258664 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:07.258635 5232 generic.go:356] "Generic (PLEG): container finished" podID="17778834-4065-463a-b5ba-b58d1d91f0d5" containerID="c17daf60b29c766d686e09128df2187bd33b5bfabd572fe4c08b967e9298772a" exitCode=0 Oct 30 18:19:07.258798 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:07.258704 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq" event={"ID":"17778834-4065-463a-b5ba-b58d1d91f0d5","Type":"ContainerDied","Data":"c17daf60b29c766d686e09128df2187bd33b5bfabd572fe4c08b967e9298772a"} Oct 30 18:19:08.009535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:08.008614 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx"] Oct 30 18:19:08.760411 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:08.760375 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq" Oct 30 18:19:08.832371 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:08.832333 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17778834-4065-463a-b5ba-b58d1d91f0d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17778834-4065-463a-b5ba-b58d1d91f0d5-maas-api-openshift-ca\") pod \"17778834-4065-463a-b5ba-b58d1d91f0d5\" (UID: \"17778834-4065-463a-b5ba-b58d1d91f0d5\") " Oct 30 18:19:08.851745 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:08.844020 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17778834-4065-463a-b5ba-b58d1d91f0d5-maas-api-openshift-ca" pod "17778834-4065-463a-b5ba-b58d1d91f0d5" (UID: "17778834-4065-463a-b5ba-b58d1d91f0d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:08.852428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:08.852035 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17778834-4065-463a-b5ba-b58d1d91f0d5-kube-api-access-bc5sp\" (UniqueName: \"kubernetes.io/projected/17778834-4065-463a-b5ba-b58d1d91f0d5-kube-api-access-bc5sp\") pod \"17778834-4065-463a-b5ba-b58d1d91f0d5\" (UID: \"17778834-4065-463a-b5ba-b58d1d91f0d5\") " Oct 30 18:19:08.852786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:08.852763 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/17778834-4065-463a-b5ba-b58d1d91f0d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17778834-4065-463a-b5ba-b58d1d91f0d5-maas-api-xks-ca\") pod \"17778834-4065-463a-b5ba-b58d1d91f0d5\" (UID: \"17778834-4065-463a-b5ba-b58d1d91f0d5\") " Oct 30 18:19:08.854393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:08.854368 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/17778834-4065-463a-b5ba-b58d1d91f0d5-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:09.039533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:09.034818 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17778834-4065-463a-b5ba-b58d1d91f0d5-kube-api-access-bc5sp" pod "17778834-4065-463a-b5ba-b58d1d91f0d5" (UID: "17778834-4065-463a-b5ba-b58d1d91f0d5"). InnerVolumeSpecName "kube-api-access-bc5sp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:09.079532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:09.077469 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc5sp\" (UniqueName: \"kubernetes.io/projected/17778834-4065-463a-b5ba-b58d1d91f0d5-kube-api-access-bc5sp\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:09.105533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:09.088227 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/17778834-4065-463a-b5ba-b58d1d91f0d5-maas-api-xks-ca" pod "17778834-4065-463a-b5ba-b58d1d91f0d5" (UID: "17778834-4065-463a-b5ba-b58d1d91f0d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:09.181613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:09.181187 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/17778834-4065-463a-b5ba-b58d1d91f0d5-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:09.290481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:09.290443 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq" Oct 30 18:19:09.291144 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:09.290489 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq" event={"ID":"17778834-4065-463a-b5ba-b58d1d91f0d5","Type":"ContainerDied","Data":"383c089d3e04a9be6767afe08f1eb663560d09c4c9303904f93ae95744928b51"} Oct 30 18:19:09.291303 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:09.291170 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="383c089d3e04a9be6767afe08f1eb663560d09c4c9303904f93ae95744928b51" Oct 30 18:19:11.208981 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:11.208948 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2"] Oct 30 18:19:11.220556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:11.220527 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-535b743b-l62q2"] Oct 30 18:19:11.383884 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:11.383851 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w"] Oct 30 18:19:11.384404 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:11.384375 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" podUID="3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" containerName="istio-proxy" containerID="cri-o://6562083aeef1ca9679b8d96bd3d803973bc6c476f775b6c8a4f7ca9d4ad1286a" gracePeriod=30 Oct 30 18:19:12.033888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:12.033854 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn"] Oct 30 18:19:12.039028 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:12.039007 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4f4baac0-jnrdn"] Oct 30 18:19:12.842773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:12.842736 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="992df831-f517-479f-8a7b-df3dd1cf31eb" path="/var/lib/kubelet/pods/992df831-f517-479f-8a7b-df3dd1cf31eb/volumes" Oct 30 18:19:12.845155 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:12.844042 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c75c2eac-859e-4b3e-8b85-f939190e70fc" path="/var/lib/kubelet/pods/c75c2eac-859e-4b3e-8b85-f939190e70fc/volumes" Oct 30 18:19:13.010092 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:13.009944 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx" podUID="00393083-9ee4-4a04-90bd-55e9de66d5e7" containerName="payload-processing" containerID="cri-o://42b707962a3cc5221cdc1ee06e9c217e011c8e06f33d0ebb1eb36b849f8928fd" gracePeriod=25 Oct 30 18:19:13.349433 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:13.349406 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx" Oct 30 18:19:13.434331 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:13.434236 5232 generic.go:356] "Generic (PLEG): container finished" podID="00393083-9ee4-4a04-90bd-55e9de66d5e7" containerID="42b707962a3cc5221cdc1ee06e9c217e011c8e06f33d0ebb1eb36b849f8928fd" exitCode=0 Oct 30 18:19:13.434331 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:13.434307 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx" Oct 30 18:19:13.434331 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:13.434319 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx" event={"ID":"00393083-9ee4-4a04-90bd-55e9de66d5e7","Type":"ContainerDied","Data":"42b707962a3cc5221cdc1ee06e9c217e011c8e06f33d0ebb1eb36b849f8928fd"} Oct 30 18:19:13.434693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:13.434383 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx" event={"ID":"00393083-9ee4-4a04-90bd-55e9de66d5e7","Type":"ContainerDied","Data":"5083c69588017a9017b8162bbb2e8127667c4a15a7f6fcc582822f23e2665f0e"} Oct 30 18:19:13.434693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:13.434420 5232 scope.go:122] "RemoveContainer" containerID="42b707962a3cc5221cdc1ee06e9c217e011c8e06f33d0ebb1eb36b849f8928fd" Oct 30 18:19:13.448227 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:13.448206 5232 scope.go:122] "RemoveContainer" containerID="42b707962a3cc5221cdc1ee06e9c217e011c8e06f33d0ebb1eb36b849f8928fd" Oct 30 18:19:13.448613 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:19:13.448584 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"42b707962a3cc5221cdc1ee06e9c217e011c8e06f33d0ebb1eb36b849f8928fd\": container with ID starting with 42b707962a3cc5221cdc1ee06e9c217e011c8e06f33d0ebb1eb36b849f8928fd not found: ID does not exist" containerID="42b707962a3cc5221cdc1ee06e9c217e011c8e06f33d0ebb1eb36b849f8928fd" Oct 30 18:19:13.448724 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:13.448624 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"42b707962a3cc5221cdc1ee06e9c217e011c8e06f33d0ebb1eb36b849f8928fd"} err="failed to get container status \"42b707962a3cc5221cdc1ee06e9c217e011c8e06f33d0ebb1eb36b849f8928fd\": rpc error: code = NotFound desc = could not find container \"42b707962a3cc5221cdc1ee06e9c217e011c8e06f33d0ebb1eb36b849f8928fd\": container with ID starting with 42b707962a3cc5221cdc1ee06e9c217e011c8e06f33d0ebb1eb36b849f8928fd not found: ID does not exist" Oct 30 18:19:13.492827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:13.492800 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/00393083-9ee4-4a04-90bd-55e9de66d5e7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00393083-9ee4-4a04-90bd-55e9de66d5e7-plugins-config-volume\") pod \"00393083-9ee4-4a04-90bd-55e9de66d5e7\" (UID: \"00393083-9ee4-4a04-90bd-55e9de66d5e7\") " Oct 30 18:19:13.492984 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:13.492895 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/00393083-9ee4-4a04-90bd-55e9de66d5e7-kube-api-access-pbvq6\" (UniqueName: \"kubernetes.io/projected/00393083-9ee4-4a04-90bd-55e9de66d5e7-kube-api-access-pbvq6\") pod \"00393083-9ee4-4a04-90bd-55e9de66d5e7\" (UID: \"00393083-9ee4-4a04-90bd-55e9de66d5e7\") " Oct 30 18:19:13.493161 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:13.493139 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00393083-9ee4-4a04-90bd-55e9de66d5e7-plugins-config-volume" pod "00393083-9ee4-4a04-90bd-55e9de66d5e7" (UID: "00393083-9ee4-4a04-90bd-55e9de66d5e7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:13.493731 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:13.493708 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00393083-9ee4-4a04-90bd-55e9de66d5e7-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:13.495218 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:13.495190 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00393083-9ee4-4a04-90bd-55e9de66d5e7-kube-api-access-pbvq6" pod "00393083-9ee4-4a04-90bd-55e9de66d5e7" (UID: "00393083-9ee4-4a04-90bd-55e9de66d5e7"). InnerVolumeSpecName "kube-api-access-pbvq6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:13.596263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:13.596215 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pbvq6\" (UniqueName: \"kubernetes.io/projected/00393083-9ee4-4a04-90bd-55e9de66d5e7-kube-api-access-pbvq6\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:13.784005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:13.783968 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx"] Oct 30 18:19:13.795498 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:13.795470 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-7cb9c50e-68dd7c7db6-k5dqx"] Oct 30 18:19:14.815766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:14.815730 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="00393083-9ee4-4a04-90bd-55e9de66d5e7" path="/var/lib/kubelet/pods/00393083-9ee4-4a04-90bd-55e9de66d5e7/volumes" Oct 30 18:19:16.776614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.776587 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:19:16.815967 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.815940 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-credential-socket\") pod \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " Oct 30 18:19:16.816157 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.815997 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-data\") pod \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " Oct 30 18:19:16.816157 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.816026 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-workload-certs\") pod \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " Oct 30 18:19:16.816157 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.816069 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-envoy\") pod \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " Oct 30 18:19:16.816322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.816174 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-podinfo\") pod \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " Oct 30 18:19:16.816322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.816276 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istiod-ca-cert\") pod \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " Oct 30 18:19:16.816322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.816306 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-token\" (UniqueName: \"kubernetes.io/projected/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-token\") pod \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " Oct 30 18:19:16.816478 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.816335 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-credential-socket" pod "3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" (UID: "3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:16.816478 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.816344 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-kube-api-access-6pjkr\" (UniqueName: \"kubernetes.io/projected/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-kube-api-access-6pjkr\") pod \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " Oct 30 18:19:16.816478 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.816471 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-workload-certs" pod "3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" (UID: "3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:16.816683 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.816583 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-workload-socket\") pod \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\" (UID: \"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984\") " Oct 30 18:19:16.817576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.816803 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-workload-socket" pod "3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" (UID: "3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:16.817576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.816879 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-data" pod "3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" (UID: "3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:16.817576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.817074 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istiod-ca-cert" pod "3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" (UID: "3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:16.817576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.817436 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:16.817576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.817459 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:16.817576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.817527 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:16.817576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.817542 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:16.817576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.817558 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:16.819363 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.819335 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-token" pod "3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" (UID: "3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:16.819749 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.819728 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-podinfo" pod "3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" (UID: "3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:19:16.819887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.819864 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-kube-api-access-6pjkr" pod "3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" (UID: "3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984"). InnerVolumeSpecName "kube-api-access-6pjkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:16.820069 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.820052 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-envoy" pod "3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" (UID: "3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:16.896583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.896547 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp"] Oct 30 18:19:16.897724 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.897698 5232 cpu_manager.go:414] "containerMap: removing container" podUID="17778834-4065-463a-b5ba-b58d1d91f0d5" containerName="revoke-keys" Oct 30 18:19:16.897724 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.897722 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17778834-4065-463a-b5ba-b58d1d91f0d5" containerName="revoke-keys" Oct 30 18:19:16.897926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.897767 5232 cpu_manager.go:414] "containerMap: removing container" podUID="3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" containerName="istio-proxy" Oct 30 18:19:16.897926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.897776 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" containerName="istio-proxy" Oct 30 18:19:16.897926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.897805 5232 cpu_manager.go:414] "containerMap: removing container" podUID="3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" containerName="istio-proxy" Oct 30 18:19:16.897926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.897812 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" containerName="istio-proxy" Oct 30 18:19:16.897926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.897861 5232 cpu_manager.go:414] "containerMap: removing container" podUID="00393083-9ee4-4a04-90bd-55e9de66d5e7" containerName="payload-processing" Oct 30 18:19:16.897926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.897868 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="00393083-9ee4-4a04-90bd-55e9de66d5e7" containerName="payload-processing" Oct 30 18:19:16.898270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.898029 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" containerName="istio-proxy" Oct 30 18:19:16.898270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.898044 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" containerName="istio-proxy" Oct 30 18:19:16.898270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.898062 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="00393083-9ee4-4a04-90bd-55e9de66d5e7" containerName="payload-processing" Oct 30 18:19:16.898270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.898072 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="17778834-4065-463a-b5ba-b58d1d91f0d5" containerName="revoke-keys" Oct 30 18:19:16.902675 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.902652 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" Oct 30 18:19:16.919768 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.918586 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-b0171017-dockercfg-gprdr\"" Oct 30 18:19:16.919768 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.918789 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp"] Oct 30 18:19:16.919768 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.918939 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-b0171017\"" Oct 30 18:19:16.920866 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.920841 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:16.920866 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.920867 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:16.921050 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.920883 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6pjkr\" (UniqueName: \"kubernetes.io/projected/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-kube-api-access-6pjkr\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:16.921050 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:16.920898 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:17.151599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.140045 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430d86ee-8426-4d48-9441-19e1366fc46e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp\" (UID: \"430d86ee-8426-4d48-9441-19e1366fc46e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" Oct 30 18:19:17.151599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.140112 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jsgh2\" (UniqueName: \"kubernetes.io/projected/430d86ee-8426-4d48-9441-19e1366fc46e-kube-api-access-jsgh2\") pod \"payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp\" (UID: \"430d86ee-8426-4d48-9441-19e1366fc46e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" Oct 30 18:19:17.243544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.229625 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr"] Oct 30 18:19:17.255552 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.253533 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430d86ee-8426-4d48-9441-19e1366fc46e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp\" (UID: \"430d86ee-8426-4d48-9441-19e1366fc46e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" Oct 30 18:19:17.255552 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.253605 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jsgh2\" (UniqueName: \"kubernetes.io/projected/430d86ee-8426-4d48-9441-19e1366fc46e-kube-api-access-jsgh2\") pod \"payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp\" (UID: \"430d86ee-8426-4d48-9441-19e1366fc46e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" Oct 30 18:19:17.334543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.310531 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430d86ee-8426-4d48-9441-19e1366fc46e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp\" (UID: \"430d86ee-8426-4d48-9441-19e1366fc46e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" Oct 30 18:19:17.344291 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.342436 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr" Oct 30 18:19:17.400646 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.391585 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77-plugins-config-volume\") pod \"payload-processing-e2e-ait-b0171017-8fc874664-qpttr\" (UID: \"1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77\") " pod="openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr" Oct 30 18:19:17.400646 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.394013 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jsgh2\" (UniqueName: \"kubernetes.io/projected/430d86ee-8426-4d48-9441-19e1366fc46e-kube-api-access-jsgh2\") pod \"payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp\" (UID: \"430d86ee-8426-4d48-9441-19e1366fc46e\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" Oct 30 18:19:17.400646 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.400614 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z46h6\" (UniqueName: \"kubernetes.io/projected/1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77-kube-api-access-z46h6\") pod \"payload-processing-e2e-ait-b0171017-8fc874664-qpttr\" (UID: \"1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77\") " pod="openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr" Oct 30 18:19:17.432487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.432348 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x"] Oct 30 18:19:17.444071 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.443971 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr"] Oct 30 18:19:17.483908 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.483851 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f84cf0c4-gkt8x"] Oct 30 18:19:17.502253 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.502223 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77-plugins-config-volume\") pod \"payload-processing-e2e-ait-b0171017-8fc874664-qpttr\" (UID: \"1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77\") " pod="openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr" Oct 30 18:19:17.502467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.502453 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z46h6\" (UniqueName: \"kubernetes.io/projected/1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77-kube-api-access-z46h6\") pod \"payload-processing-e2e-ait-b0171017-8fc874664-qpttr\" (UID: \"1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77\") " pod="openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr" Oct 30 18:19:17.503107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.503014 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77-plugins-config-volume\") pod \"payload-processing-e2e-ait-b0171017-8fc874664-qpttr\" (UID: \"1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77\") " pod="openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr" Oct 30 18:19:17.522087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.522056 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z46h6\" (UniqueName: \"kubernetes.io/projected/1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77-kube-api-access-z46h6\") pod \"payload-processing-e2e-ait-b0171017-8fc874664-qpttr\" (UID: \"1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77\") " pod="openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr" Oct 30 18:19:17.522417 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.522396 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" Oct 30 18:19:17.542910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.542878 5232 generic.go:356] "Generic (PLEG): container finished" podID="3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" containerID="6562083aeef1ca9679b8d96bd3d803973bc6c476f775b6c8a4f7ca9d4ad1286a" exitCode=0 Oct 30 18:19:17.543083 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.542946 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" Oct 30 18:19:17.543083 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.543028 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" event={"ID":"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984","Type":"ContainerDied","Data":"6562083aeef1ca9679b8d96bd3d803973bc6c476f775b6c8a4f7ca9d4ad1286a"} Oct 30 18:19:17.543083 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.543075 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w" event={"ID":"3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984","Type":"ContainerDied","Data":"725d430383e74c0649a31aaada638d950b90238bf6c83de19fefd8777e4fd0e7"} Oct 30 18:19:17.543326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.543098 5232 scope.go:122] "RemoveContainer" containerID="6562083aeef1ca9679b8d96bd3d803973bc6c476f775b6c8a4f7ca9d4ad1286a" Oct 30 18:19:17.581983 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.580242 5232 scope.go:122] "RemoveContainer" containerID="bc9065ec30dd487b69207a9afcb3e5f34eabe7897fc972d52595d651e7a70e7d" Oct 30 18:19:17.595119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.595094 5232 scope.go:122] "RemoveContainer" containerID="6562083aeef1ca9679b8d96bd3d803973bc6c476f775b6c8a4f7ca9d4ad1286a" Oct 30 18:19:17.630127 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:19:17.595670 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6562083aeef1ca9679b8d96bd3d803973bc6c476f775b6c8a4f7ca9d4ad1286a\": container with ID starting with 6562083aeef1ca9679b8d96bd3d803973bc6c476f775b6c8a4f7ca9d4ad1286a not found: ID does not exist" containerID="6562083aeef1ca9679b8d96bd3d803973bc6c476f775b6c8a4f7ca9d4ad1286a" Oct 30 18:19:17.630127 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.595712 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6562083aeef1ca9679b8d96bd3d803973bc6c476f775b6c8a4f7ca9d4ad1286a"} err="failed to get container status \"6562083aeef1ca9679b8d96bd3d803973bc6c476f775b6c8a4f7ca9d4ad1286a\": rpc error: code = NotFound desc = could not find container \"6562083aeef1ca9679b8d96bd3d803973bc6c476f775b6c8a4f7ca9d4ad1286a\": container with ID starting with 6562083aeef1ca9679b8d96bd3d803973bc6c476f775b6c8a4f7ca9d4ad1286a not found: ID does not exist" Oct 30 18:19:17.630127 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.595752 5232 scope.go:122] "RemoveContainer" containerID="bc9065ec30dd487b69207a9afcb3e5f34eabe7897fc972d52595d651e7a70e7d" Oct 30 18:19:17.982335 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.882876 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr" Oct 30 18:19:17.982335 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:19:17.917529 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc9065ec30dd487b69207a9afcb3e5f34eabe7897fc972d52595d651e7a70e7d\": container with ID starting with bc9065ec30dd487b69207a9afcb3e5f34eabe7897fc972d52595d651e7a70e7d not found: ID does not exist" containerID="bc9065ec30dd487b69207a9afcb3e5f34eabe7897fc972d52595d651e7a70e7d" Oct 30 18:19:17.982335 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:17.917589 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc9065ec30dd487b69207a9afcb3e5f34eabe7897fc972d52595d651e7a70e7d"} err="failed to get container status \"bc9065ec30dd487b69207a9afcb3e5f34eabe7897fc972d52595d651e7a70e7d\": rpc error: code = NotFound desc = could not find container \"bc9065ec30dd487b69207a9afcb3e5f34eabe7897fc972d52595d651e7a70e7d\": container with ID starting with bc9065ec30dd487b69207a9afcb3e5f34eabe7897fc972d52595d651e7a70e7d not found: ID does not exist" Oct 30 18:19:18.355070 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:19:18.355039 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod430d86ee_8426_4d48_9441_19e1366fc46e.slice/crio-63954a13ea66d517a68c8f4a7c0c8389d92bf6a8a70c10782ae6294879fbb834 WatchSource:0}: Error finding container 63954a13ea66d517a68c8f4a7c0c8389d92bf6a8a70c10782ae6294879fbb834: Status 404 returned error can't find the container with id 63954a13ea66d517a68c8f4a7c0c8389d92bf6a8a70c10782ae6294879fbb834 Oct 30 18:19:18.405791 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:18.405748 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w"] Oct 30 18:19:18.430193 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:18.430128 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-535b743b-gw-openshift-default-f97c88957-hl77w"] Oct 30 18:19:18.436275 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:18.436227 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp"] Oct 30 18:19:18.554955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:18.554918 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" event={"ID":"430d86ee-8426-4d48-9441-19e1366fc46e","Type":"ContainerStarted","Data":"21a7ffb6e8d75434e1cae83ebf66568509853cfe16e9ab7bff0faca55176b000"} Oct 30 18:19:18.555127 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:18.554983 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" event={"ID":"430d86ee-8426-4d48-9441-19e1366fc46e","Type":"ContainerStarted","Data":"63954a13ea66d517a68c8f4a7c0c8389d92bf6a8a70c10782ae6294879fbb834"} Oct 30 18:19:18.555127 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:18.555012 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" Oct 30 18:19:18.598048 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:18.597935 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" podStartSLOduration=2.597915606 podStartE2EDuration="2.597915606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:18.591805181 +0000 UTC m=+1057.908225503" watchObservedRunningTime="2026-10-30 18:19:18.597915606 +0000 UTC m=+1057.914335928" Oct 30 18:19:18.687558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:18.687500 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr"] Oct 30 18:19:18.692626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:18.692603 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-7cb9c50e-2mnsr"] Oct 30 18:19:18.702434 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:18.702410 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr"] Oct 30 18:19:18.704277 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:19:18.704252 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c0c8f9b_05b3_4ce3_bef1_2518d99a8c77.slice/crio-b1fac97be40a7c4ace47bad5d40a3d0df0206fe8559ffa5a4359c6efcc5590a9 WatchSource:0}: Error finding container b1fac97be40a7c4ace47bad5d40a3d0df0206fe8559ffa5a4359c6efcc5590a9: Status 404 returned error can't find the container with id b1fac97be40a7c4ace47bad5d40a3d0df0206fe8559ffa5a4359c6efcc5590a9 Oct 30 18:19:18.865016 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:18.862255 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f5333ad-eb5b-42e0-b748-b1b48a4c00ad" path="/var/lib/kubelet/pods/0f5333ad-eb5b-42e0-b748-b1b48a4c00ad/volumes" Oct 30 18:19:18.865016 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:18.863072 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984" path="/var/lib/kubelet/pods/3fddc7f1-0ab6-45e0-9a80-6cfbc05a4984/volumes" Oct 30 18:19:18.865016 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:18.863760 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44c9b672-70cf-40ce-aa40-f06c8462273e" path="/var/lib/kubelet/pods/44c9b672-70cf-40ce-aa40-f06c8462273e/volumes" Oct 30 18:19:18.893535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:18.891762 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d"] Oct 30 18:19:18.893535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:18.892002 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" podUID="f187b761-f4e5-4126-a03d-5f4418db9cf0" containerName="istio-proxy" containerID="cri-o://4c18324f59f7a39fc8608597903af09c1981106b77afb336ebdd50810ba9c054" gracePeriod=30 Oct 30 18:19:19.030208 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:19.029939 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf"] Oct 30 18:19:19.030208 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:19.030138 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" podUID="fe9253e4-0ac7-47ba-85d8-b55a75b5334b" containerName="istio-proxy" containerID="cri-o://a18a817d9325883ec44f6b37b9d69951e3e0718dfb0ed51a20ddab2d78578881" gracePeriod=30 Oct 30 18:19:19.567551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:19.567480 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr" event={"ID":"1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77","Type":"ContainerStarted","Data":"c95c0a7694561d638bf94aec8bc9367fd8a39bce884c660c9f13ce2c8f575f57"} Oct 30 18:19:19.567551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:19.567555 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr" event={"ID":"1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77","Type":"ContainerStarted","Data":"b1fac97be40a7c4ace47bad5d40a3d0df0206fe8559ffa5a4359c6efcc5590a9"} Oct 30 18:19:19.567864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:19.567847 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr" Oct 30 18:19:19.608406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:19.608342 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr" podStartSLOduration=3.608322896 podStartE2EDuration="3.608322896s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:19.606061267 +0000 UTC m=+1058.922481612" watchObservedRunningTime="2026-10-30 18:19:19.608322896 +0000 UTC m=+1058.924743218" Oct 30 18:19:20.685525 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:20.685471 5232 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:19:20.685972 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:20.685560 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" podUID="f187b761-f4e5-4126-a03d-5f4418db9cf0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:19:20.958657 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:20.949987 5232 patch_prober.go:29] interesting pod/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:19:20.958657 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:20.950042 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" podUID="fe9253e4-0ac7-47ba-85d8-b55a75b5334b" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:19:25.060424 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.060204 5232 generic.go:356] "Generic (PLEG): container finished" podID="f187b761-f4e5-4126-a03d-5f4418db9cf0" containerID="4c18324f59f7a39fc8608597903af09c1981106b77afb336ebdd50810ba9c054" exitCode=0 Oct 30 18:19:25.061226 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.061016 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" event={"ID":"f187b761-f4e5-4126-a03d-5f4418db9cf0","Type":"ContainerDied","Data":"4c18324f59f7a39fc8608597903af09c1981106b77afb336ebdd50810ba9c054"} Oct 30 18:19:25.062430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.062378 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq"] Oct 30 18:19:25.071717 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.071686 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-f6a224e7-twwxq"] Oct 30 18:19:25.085955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.085774 5232 generic.go:356] "Generic (PLEG): container finished" podID="fe9253e4-0ac7-47ba-85d8-b55a75b5334b" containerID="a18a817d9325883ec44f6b37b9d69951e3e0718dfb0ed51a20ddab2d78578881" exitCode=0 Oct 30 18:19:25.085955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.085863 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" event={"ID":"fe9253e4-0ac7-47ba-85d8-b55a75b5334b","Type":"ContainerDied","Data":"a18a817d9325883ec44f6b37b9d69951e3e0718dfb0ed51a20ddab2d78578881"} Oct 30 18:19:25.162241 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.162213 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:19:25.230799 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.230774 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:19:25.359542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.348168 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-kube-api-access-rcgps\" (UniqueName: \"kubernetes.io/projected/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-kube-api-access-rcgps\") pod \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " Oct 30 18:19:25.359542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.348398 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-envoy\") pod \"f187b761-f4e5-4126-a03d-5f4418db9cf0\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " Oct 30 18:19:25.359542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.348435 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-credential-socket\") pod \"f187b761-f4e5-4126-a03d-5f4418db9cf0\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " Oct 30 18:19:25.359542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.348487 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-token\" (UniqueName: \"kubernetes.io/projected/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-token\") pod \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " Oct 30 18:19:25.359542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.348542 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-token\" (UniqueName: \"kubernetes.io/projected/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-token\") pod \"f187b761-f4e5-4126-a03d-5f4418db9cf0\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " Oct 30 18:19:25.359542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.348597 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-workload-certs\") pod \"f187b761-f4e5-4126-a03d-5f4418db9cf0\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " Oct 30 18:19:25.359542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.348628 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-data\") pod \"f187b761-f4e5-4126-a03d-5f4418db9cf0\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " Oct 30 18:19:25.359542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.348745 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-workload-certs\") pod \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " Oct 30 18:19:25.359542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.348793 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-envoy\") pod \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " Oct 30 18:19:25.359542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.348840 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f187b761-f4e5-4126-a03d-5f4418db9cf0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f187b761-f4e5-4126-a03d-5f4418db9cf0-istiod-ca-cert\") pod \"f187b761-f4e5-4126-a03d-5f4418db9cf0\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " Oct 30 18:19:25.359542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.348949 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-podinfo\") pod \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " Oct 30 18:19:25.359542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.349033 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-podinfo\") pod \"f187b761-f4e5-4126-a03d-5f4418db9cf0\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " Oct 30 18:19:25.359542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.349059 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-workload-socket\") pod \"f187b761-f4e5-4126-a03d-5f4418db9cf0\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " Oct 30 18:19:25.359542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.349092 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-data\") pod \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " Oct 30 18:19:25.359542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.349119 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-workload-socket\") pod \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " Oct 30 18:19:25.359542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.349150 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-credential-socket\") pod \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " Oct 30 18:19:25.359542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.349176 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istiod-ca-cert\") pod \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\" (UID: \"fe9253e4-0ac7-47ba-85d8-b55a75b5334b\") " Oct 30 18:19:25.359542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.349227 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f187b761-f4e5-4126-a03d-5f4418db9cf0-kube-api-access-4sdsz\" (UniqueName: \"kubernetes.io/projected/f187b761-f4e5-4126-a03d-5f4418db9cf0-kube-api-access-4sdsz\") pod \"f187b761-f4e5-4126-a03d-5f4418db9cf0\" (UID: \"f187b761-f4e5-4126-a03d-5f4418db9cf0\") " Oct 30 18:19:25.388673 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.365099 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-workload-certs" pod "fe9253e4-0ac7-47ba-85d8-b55a75b5334b" (UID: "fe9253e4-0ac7-47ba-85d8-b55a75b5334b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:25.388673 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.383310 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-credential-socket" pod "fe9253e4-0ac7-47ba-85d8-b55a75b5334b" (UID: "fe9253e4-0ac7-47ba-85d8-b55a75b5334b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:25.410742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.409573 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-data" pod "fe9253e4-0ac7-47ba-85d8-b55a75b5334b" (UID: "fe9253e4-0ac7-47ba-85d8-b55a75b5334b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:25.410742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.410309 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f187b761-f4e5-4126-a03d-5f4418db9cf0-istiod-ca-cert" pod "f187b761-f4e5-4126-a03d-5f4418db9cf0" (UID: "f187b761-f4e5-4126-a03d-5f4418db9cf0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:25.425533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.420880 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-workload-certs" pod "f187b761-f4e5-4126-a03d-5f4418db9cf0" (UID: "f187b761-f4e5-4126-a03d-5f4418db9cf0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:25.425533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.422637 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istiod-ca-cert" pod "fe9253e4-0ac7-47ba-85d8-b55a75b5334b" (UID: "fe9253e4-0ac7-47ba-85d8-b55a75b5334b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:25.425533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.422776 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-token" pod "fe9253e4-0ac7-47ba-85d8-b55a75b5334b" (UID: "fe9253e4-0ac7-47ba-85d8-b55a75b5334b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:25.425533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.423024 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-data" pod "f187b761-f4e5-4126-a03d-5f4418db9cf0" (UID: "f187b761-f4e5-4126-a03d-5f4418db9cf0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:25.425533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.424316 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-credential-socket" pod "f187b761-f4e5-4126-a03d-5f4418db9cf0" (UID: "f187b761-f4e5-4126-a03d-5f4418db9cf0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:25.427268 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.427194 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-podinfo" pod "fe9253e4-0ac7-47ba-85d8-b55a75b5334b" (UID: "fe9253e4-0ac7-47ba-85d8-b55a75b5334b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:19:25.436792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.430270 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-workload-socket" pod "fe9253e4-0ac7-47ba-85d8-b55a75b5334b" (UID: "fe9253e4-0ac7-47ba-85d8-b55a75b5334b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:25.437067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.431095 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-workload-socket" pod "f187b761-f4e5-4126-a03d-5f4418db9cf0" (UID: "f187b761-f4e5-4126-a03d-5f4418db9cf0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:25.437067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.434580 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-token" pod "f187b761-f4e5-4126-a03d-5f4418db9cf0" (UID: "f187b761-f4e5-4126-a03d-5f4418db9cf0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:25.437067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.434728 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f187b761-f4e5-4126-a03d-5f4418db9cf0-kube-api-access-4sdsz" pod "f187b761-f4e5-4126-a03d-5f4418db9cf0" (UID: "f187b761-f4e5-4126-a03d-5f4418db9cf0"). InnerVolumeSpecName "kube-api-access-4sdsz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:25.437321 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.434769 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-kube-api-access-rcgps" pod "fe9253e4-0ac7-47ba-85d8-b55a75b5334b" (UID: "fe9253e4-0ac7-47ba-85d8-b55a75b5334b"). InnerVolumeSpecName "kube-api-access-rcgps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:25.437321 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.436173 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-envoy" pod "fe9253e4-0ac7-47ba-85d8-b55a75b5334b" (UID: "fe9253e4-0ac7-47ba-85d8-b55a75b5334b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:25.437321 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.436816 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-envoy" pod "f187b761-f4e5-4126-a03d-5f4418db9cf0" (UID: "f187b761-f4e5-4126-a03d-5f4418db9cf0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:25.437321 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.437249 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-podinfo" pod "f187b761-f4e5-4126-a03d-5f4418db9cf0" (UID: "f187b761-f4e5-4126-a03d-5f4418db9cf0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:19:25.451460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.451383 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.451460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.451415 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.451460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.451429 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.451460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.451440 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.451460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.451450 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.451460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.451461 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.451817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.451472 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f187b761-f4e5-4126-a03d-5f4418db9cf0-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.451817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.451481 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.451817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.451489 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.451817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.451499 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.451817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.451528 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.451817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.451536 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.451817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.451547 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.451817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.451556 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.451817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.451564 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sdsz\" (UniqueName: \"kubernetes.io/projected/f187b761-f4e5-4126-a03d-5f4418db9cf0-kube-api-access-4sdsz\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.451817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.451585 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rcgps\" (UniqueName: \"kubernetes.io/projected/fe9253e4-0ac7-47ba-85d8-b55a75b5334b-kube-api-access-rcgps\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.451817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.451595 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:25.451817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:25.451604 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f187b761-f4e5-4126-a03d-5f4418db9cf0-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:26.096663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:26.096635 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" Oct 30 18:19:26.097128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:26.096629 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf" event={"ID":"fe9253e4-0ac7-47ba-85d8-b55a75b5334b","Type":"ContainerDied","Data":"2d1f7e593a64164bb55ee32a2fd69486f0476ce833b62d8fbf86066671297459"} Oct 30 18:19:26.097128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:26.096773 5232 scope.go:122] "RemoveContainer" containerID="a18a817d9325883ec44f6b37b9d69951e3e0718dfb0ed51a20ddab2d78578881" Oct 30 18:19:26.098499 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:26.098473 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" event={"ID":"f187b761-f4e5-4126-a03d-5f4418db9cf0","Type":"ContainerDied","Data":"a26734e2c09643b3d0d08bf13f218836c01b99772ff53c8e77328e3e8ff5d0d1"} Oct 30 18:19:26.098499 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:26.098489 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d" Oct 30 18:19:26.109851 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:26.109828 5232 scope.go:122] "RemoveContainer" containerID="4c18324f59f7a39fc8608597903af09c1981106b77afb336ebdd50810ba9c054" Oct 30 18:19:26.150714 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:26.150683 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d"] Oct 30 18:19:26.162654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:26.162630 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-bfb8d"] Oct 30 18:19:26.187495 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:26.187468 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf"] Oct 30 18:19:26.192768 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:26.192740 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-7cb9c50e-openshift-default-6995c464bf-stmhf"] Oct 30 18:19:26.824536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:26.824084 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17778834-4065-463a-b5ba-b58d1d91f0d5" path="/var/lib/kubelet/pods/17778834-4065-463a-b5ba-b58d1d91f0d5/volumes" Oct 30 18:19:26.828530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:26.825016 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f187b761-f4e5-4126-a03d-5f4418db9cf0" path="/var/lib/kubelet/pods/f187b761-f4e5-4126-a03d-5f4418db9cf0/volumes" Oct 30 18:19:26.846534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:26.833165 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe9253e4-0ac7-47ba-85d8-b55a75b5334b" path="/var/lib/kubelet/pods/fe9253e4-0ac7-47ba-85d8-b55a75b5334b/volumes" Oct 30 18:19:27.035003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:27.034970 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp"] Oct 30 18:19:27.037614 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:19:27.037580 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:19:27.045743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:27.045710 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d8c58813-569dp"] Oct 30 18:19:28.973537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:28.894385 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94fd2aa8-49b1-432b-9adf-2381ff3451b0" path="/var/lib/kubelet/pods/94fd2aa8-49b1-432b-9adf-2381ff3451b0/volumes" Oct 30 18:19:28.995776 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:28.995734 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:19:29.163535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:29.130613 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:19:29.379537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:29.375639 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:19:29.393196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:29.390366 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:19:29.571728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:29.571694 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" Oct 30 18:19:30.580491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:30.580464 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr" Oct 30 18:19:35.596680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:35.596630 5232 patch_prober.go:29] interesting pod/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:19:35.597105 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:35.596693 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" podUID="19303a05-780a-4c03-9df6-2db3aeee3a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:19:36.365943 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.365368 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g"] Oct 30 18:19:36.372780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.372742 5232 cpu_manager.go:414] "containerMap: removing container" podUID="f187b761-f4e5-4126-a03d-5f4418db9cf0" containerName="istio-proxy" Oct 30 18:19:36.372780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.372774 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f187b761-f4e5-4126-a03d-5f4418db9cf0" containerName="istio-proxy" Oct 30 18:19:36.373002 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.372813 5232 cpu_manager.go:414] "containerMap: removing container" podUID="fe9253e4-0ac7-47ba-85d8-b55a75b5334b" containerName="istio-proxy" Oct 30 18:19:36.373002 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.372821 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe9253e4-0ac7-47ba-85d8-b55a75b5334b" containerName="istio-proxy" Oct 30 18:19:36.373218 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.373195 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="f187b761-f4e5-4126-a03d-5f4418db9cf0" containerName="istio-proxy" Oct 30 18:19:36.373258 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.373226 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe9253e4-0ac7-47ba-85d8-b55a75b5334b" containerName="istio-proxy" Oct 30 18:19:36.379768 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.379738 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g" Oct 30 18:19:36.383136 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.383103 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g"] Oct 30 18:19:36.394818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.394788 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:19:36.406925 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.406899 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:19:36.491984 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.491953 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hz8rg\" (UniqueName: \"kubernetes.io/projected/6bbacde3-9498-4474-a6f1-7647c1392f9f-kube-api-access-hz8rg\") pod \"maas-api-revoke-keys-e2e-derive-a90958fa-26q5g\" (UID: \"6bbacde3-9498-4474-a6f1-7647c1392f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g" Oct 30 18:19:36.492154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.492108 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbacde3-9498-4474-a6f1-7647c1392f9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a90958fa-26q5g\" (UID: \"6bbacde3-9498-4474-a6f1-7647c1392f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g" Oct 30 18:19:36.492218 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.492202 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbacde3-9498-4474-a6f1-7647c1392f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a90958fa-26q5g\" (UID: \"6bbacde3-9498-4474-a6f1-7647c1392f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g" Oct 30 18:19:36.594074 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.594037 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hz8rg\" (UniqueName: \"kubernetes.io/projected/6bbacde3-9498-4474-a6f1-7647c1392f9f-kube-api-access-hz8rg\") pod \"maas-api-revoke-keys-e2e-derive-a90958fa-26q5g\" (UID: \"6bbacde3-9498-4474-a6f1-7647c1392f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g" Oct 30 18:19:36.594244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.594171 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbacde3-9498-4474-a6f1-7647c1392f9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a90958fa-26q5g\" (UID: \"6bbacde3-9498-4474-a6f1-7647c1392f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g" Oct 30 18:19:36.594293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.594269 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbacde3-9498-4474-a6f1-7647c1392f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a90958fa-26q5g\" (UID: \"6bbacde3-9498-4474-a6f1-7647c1392f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g" Oct 30 18:19:36.595139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.595114 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbacde3-9498-4474-a6f1-7647c1392f9f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-a90958fa-26q5g\" (UID: \"6bbacde3-9498-4474-a6f1-7647c1392f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g" Oct 30 18:19:36.596764 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.596744 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbacde3-9498-4474-a6f1-7647c1392f9f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-a90958fa-26q5g\" (UID: \"6bbacde3-9498-4474-a6f1-7647c1392f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g" Oct 30 18:19:36.613691 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.613658 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hz8rg\" (UniqueName: \"kubernetes.io/projected/6bbacde3-9498-4474-a6f1-7647c1392f9f-kube-api-access-hz8rg\") pod \"maas-api-revoke-keys-e2e-derive-a90958fa-26q5g\" (UID: \"6bbacde3-9498-4474-a6f1-7647c1392f9f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g" Oct 30 18:19:36.700672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:36.700597 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g" Oct 30 18:19:37.003668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:37.003641 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g"] Oct 30 18:19:37.005814 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:19:37.005783 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bbacde3_9498_4474_a6f1_7647c1392f9f.slice/crio-3ffe935ec55f011369fc7eddf8f42450e4da2307ad83b86405f1d10e3b37df1a WatchSource:0}: Error finding container 3ffe935ec55f011369fc7eddf8f42450e4da2307ad83b86405f1d10e3b37df1a: Status 404 returned error can't find the container with id 3ffe935ec55f011369fc7eddf8f42450e4da2307ad83b86405f1d10e3b37df1a Oct 30 18:19:37.643536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:37.631644 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g" event={"ID":"6bbacde3-9498-4474-a6f1-7647c1392f9f","Type":"ContainerStarted","Data":"03f0b05d7e72dbc45e263c587fb1bba281dac4b35395432219db05780e4b0b90"} Oct 30 18:19:37.643536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:37.631718 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g" event={"ID":"6bbacde3-9498-4474-a6f1-7647c1392f9f","Type":"ContainerStarted","Data":"3ffe935ec55f011369fc7eddf8f42450e4da2307ad83b86405f1d10e3b37df1a"} Oct 30 18:19:37.705584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:37.696131 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g" podStartSLOduration=1.696105472 podStartE2EDuration="1.696105472s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:37.682274389 +0000 UTC m=+1076.998694713" watchObservedRunningTime="2026-10-30 18:19:37.696105472 +0000 UTC m=+1077.012525794" Oct 30 18:19:38.693531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:38.688413 5232 generic.go:356] "Generic (PLEG): container finished" podID="6bbacde3-9498-4474-a6f1-7647c1392f9f" containerID="03f0b05d7e72dbc45e263c587fb1bba281dac4b35395432219db05780e4b0b90" exitCode=0 Oct 30 18:19:38.693531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:38.688615 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g" event={"ID":"6bbacde3-9498-4474-a6f1-7647c1392f9f","Type":"ContainerDied","Data":"03f0b05d7e72dbc45e263c587fb1bba281dac4b35395432219db05780e4b0b90"} Oct 30 18:19:38.704472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:38.701451 5232 generic.go:356] "Generic (PLEG): container finished" podID="19303a05-780a-4c03-9df6-2db3aeee3a9b" containerID="437a7ca20c9e513770e6ef040e776c44c14c94542879f7b5613757bb47bbcdcf" exitCode=0 Oct 30 18:19:38.704472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:38.702144 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" event={"ID":"19303a05-780a-4c03-9df6-2db3aeee3a9b","Type":"ContainerDied","Data":"437a7ca20c9e513770e6ef040e776c44c14c94542879f7b5613757bb47bbcdcf"} Oct 30 18:19:38.704472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:38.702737 5232 scope.go:122] "RemoveContainer" containerID="437a7ca20c9e513770e6ef040e776c44c14c94542879f7b5613757bb47bbcdcf" Oct 30 18:19:38.707641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:38.706567 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:19:38.707641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:38.706681 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:19:38.707641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:38.706727 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:19:38.820134 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:38.819464 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4"] Oct 30 18:19:38.834771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:38.831326 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:38.847360 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:38.847323 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4"] Oct 30 18:19:38.969906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:38.969824 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:38.970077 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:38.969915 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:38.970168 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:38.970141 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:38.970236 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:38.970209 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:38.970385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:38.970365 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:38.970544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:38.970491 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fr8w7\" (UniqueName: \"kubernetes.io/projected/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-kube-api-access-fr8w7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:38.970597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:38.970562 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:39.074126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.072889 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:39.074126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.073050 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:39.074126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.073157 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fr8w7\" (UniqueName: \"kubernetes.io/projected/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-kube-api-access-fr8w7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:39.074126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.073206 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:39.074126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.073267 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:39.074126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.073325 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:39.074126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.073524 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:39.079557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.079495 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:39.084550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.083710 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:39.084550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.084022 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:39.084550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.084397 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:39.084818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.084738 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:39.094418 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.094383 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fr8w7\" (UniqueName: \"kubernetes.io/projected/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-kube-api-access-fr8w7\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:39.094722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.094702 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-2s5j4\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:39.161776 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.161746 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:39.663687 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.662330 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4"] Oct 30 18:19:39.676578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.675850 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:19:39.676578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.675942 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:19:39.730367 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.730333 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" event={"ID":"19303a05-780a-4c03-9df6-2db3aeee3a9b","Type":"ContainerStarted","Data":"6a1a8c80a4f4e9ea5a7b6234181db2931b6b2977b6f5c5ec6a3a16faa63e7bca"} Oct 30 18:19:39.734379 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:39.734297 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" event={"ID":"29a994e2-22c0-49ce-8d6b-1898bf2f7edc","Type":"ContainerStarted","Data":"7000becc2b9b477c577416069f5745690fc85473facedff80bb31d387a8d8ee1"} Oct 30 18:19:40.002975 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.002952 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g" Oct 30 18:19:40.050584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.050553 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6bbacde3-9498-4474-a6f1-7647c1392f9f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbacde3-9498-4474-a6f1-7647c1392f9f-maas-api-xks-ca\") pod \"6bbacde3-9498-4474-a6f1-7647c1392f9f\" (UID: \"6bbacde3-9498-4474-a6f1-7647c1392f9f\") " Oct 30 18:19:40.050785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.050744 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bbacde3-9498-4474-a6f1-7647c1392f9f-kube-api-access-hz8rg\" (UniqueName: \"kubernetes.io/projected/6bbacde3-9498-4474-a6f1-7647c1392f9f-kube-api-access-hz8rg\") pod \"6bbacde3-9498-4474-a6f1-7647c1392f9f\" (UID: \"6bbacde3-9498-4474-a6f1-7647c1392f9f\") " Oct 30 18:19:40.050856 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.050823 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bbacde3-9498-4474-a6f1-7647c1392f9f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbacde3-9498-4474-a6f1-7647c1392f9f-maas-api-openshift-ca\") pod \"6bbacde3-9498-4474-a6f1-7647c1392f9f\" (UID: \"6bbacde3-9498-4474-a6f1-7647c1392f9f\") " Oct 30 18:19:40.052271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.052228 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bbacde3-9498-4474-a6f1-7647c1392f9f-maas-api-openshift-ca" pod "6bbacde3-9498-4474-a6f1-7647c1392f9f" (UID: "6bbacde3-9498-4474-a6f1-7647c1392f9f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:40.059177 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.059152 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bbacde3-9498-4474-a6f1-7647c1392f9f-kube-api-access-hz8rg" pod "6bbacde3-9498-4474-a6f1-7647c1392f9f" (UID: "6bbacde3-9498-4474-a6f1-7647c1392f9f"). InnerVolumeSpecName "kube-api-access-hz8rg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:40.061193 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.061157 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6bbacde3-9498-4474-a6f1-7647c1392f9f-maas-api-xks-ca" pod "6bbacde3-9498-4474-a6f1-7647c1392f9f" (UID: "6bbacde3-9498-4474-a6f1-7647c1392f9f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:40.153214 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.153171 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hz8rg\" (UniqueName: \"kubernetes.io/projected/6bbacde3-9498-4474-a6f1-7647c1392f9f-kube-api-access-hz8rg\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:40.153214 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.153214 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6bbacde3-9498-4474-a6f1-7647c1392f9f-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:40.153473 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.153230 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6bbacde3-9498-4474-a6f1-7647c1392f9f-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:40.579225 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.579189 5232 patch_prober.go:29] interesting pod/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 18:19:40.579554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.579215 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:19:40.579707 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.579502 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" podUID="19303a05-780a-4c03-9df6-2db3aeee3a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 18:19:40.754996 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.754736 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g" event={"ID":"6bbacde3-9498-4474-a6f1-7647c1392f9f","Type":"ContainerDied","Data":"3ffe935ec55f011369fc7eddf8f42450e4da2307ad83b86405f1d10e3b37df1a"} Oct 30 18:19:40.754996 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.754790 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ffe935ec55f011369fc7eddf8f42450e4da2307ad83b86405f1d10e3b37df1a" Oct 30 18:19:40.754996 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.754869 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g" Oct 30 18:19:40.758742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.758697 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" event={"ID":"29a994e2-22c0-49ce-8d6b-1898bf2f7edc","Type":"ContainerStarted","Data":"e8879e631e66c9aec5f31638b7c251c79ab3bdeff767bce50b0940038e90ee0c"} Oct 30 18:19:40.759155 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.759109 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:40.762059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.762035 5232 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-2s5j4 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:19:40.762256 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.762238 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" podUID="29a994e2-22c0-49ce-8d6b-1898bf2f7edc" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:19:40.805784 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:40.805724 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" podStartSLOduration=2.805706298 podStartE2EDuration="2.805706298s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:19:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:19:40.798782062 +0000 UTC m=+1080.115202385" watchObservedRunningTime="2026-10-30 18:19:40.805706298 +0000 UTC m=+1080.122126620" Oct 30 18:19:41.579158 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:41.579119 5232 patch_prober.go:29] interesting pod/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 18:19:41.579344 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:41.579166 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" podUID="19303a05-780a-4c03-9df6-2db3aeee3a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 18:19:41.771950 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:41.771919 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:19:42.578862 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:42.578818 5232 patch_prober.go:29] interesting pod/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 18:19:42.579076 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:42.578873 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" podUID="19303a05-780a-4c03-9df6-2db3aeee3a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 18:19:43.578319 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:43.578286 5232 patch_prober.go:29] interesting pod/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 18:19:43.578770 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:43.578334 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" podUID="19303a05-780a-4c03-9df6-2db3aeee3a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 18:19:44.578489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:44.578451 5232 patch_prober.go:29] interesting pod/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 18:19:44.578933 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:44.578542 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" podUID="19303a05-780a-4c03-9df6-2db3aeee3a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 18:19:45.590078 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:45.590039 5232 patch_prober.go:29] interesting pod/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" start-of-body= Oct 30 18:19:45.590675 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:45.590642 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" podUID="19303a05-780a-4c03-9df6-2db3aeee3a9b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.134:15021/healthz/ready\": dial tcp 10.133.0.134:15021: connect: connection refused" Oct 30 18:19:46.011118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:46.011073 5232 scope.go:122] "RemoveContainer" containerID="1b0828f41bdad4df2e06bfc24fe7f9ab37044d52a7b52fd49f969f68a1b82b39" Oct 30 18:19:46.028573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:46.028551 5232 scope.go:122] "RemoveContainer" containerID="da5858690e7aa92f06d81b6f973e5cdd3c19532b948c38370221afc607425432" Oct 30 18:19:46.042746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:46.042719 5232 scope.go:122] "RemoveContainer" containerID="1cf92e3b9e0067347d91f3010722fd5c475cecc314cb9909f6e32dea97744d46" Oct 30 18:19:46.058366 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:46.058340 5232 scope.go:122] "RemoveContainer" containerID="fdffb323bd05e3cdf5847972c2838c829b94871990cb9cad125a56e6dcf8d952" Oct 30 18:19:46.072618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:46.072592 5232 scope.go:122] "RemoveContainer" containerID="7b1b1b42b07d3e317bd6a206330064e83f81cc8b52cce213bc8f94b1082b0353" Oct 30 18:19:46.084766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:46.084744 5232 scope.go:122] "RemoveContainer" containerID="7e5d0e7be3a7a2f28b205921fc4b9806ac6db68d0544fcf0831d85369d548c17" Oct 30 18:19:46.095414 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:46.095387 5232 scope.go:122] "RemoveContainer" containerID="a7bd44c1c0fa0746da65b5f1f874af07a9903f1c66266215343f71f867dcc6fa" Oct 30 18:19:46.109616 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:46.109591 5232 scope.go:122] "RemoveContainer" containerID="34a2903fed7c330ca21c07983552726aa0b1d00fea647b0223459d789909d6ec" Oct 30 18:19:46.121115 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:46.121093 5232 scope.go:122] "RemoveContainer" containerID="ff7f4cabfe259ee6b4131e4b515e509eb4a0aad1c74298af92fc504f4fe8d915" Oct 30 18:19:46.131248 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:46.131229 5232 scope.go:122] "RemoveContainer" containerID="2217292207ec98fb25c87a4f740ecae77cc128cd2c9b9136884c54e2fda78862" Oct 30 18:19:46.140775 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:46.140759 5232 scope.go:122] "RemoveContainer" containerID="c44a0a10e7651b3d5c3534aaf56f82c97e9413521dc9874420f5f78ed2486903" Oct 30 18:19:46.152088 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:46.151982 5232 scope.go:122] "RemoveContainer" containerID="3c182b65baf88209cab82f401db76605162ac7bc4c885c18580438afdd1a7f8d" Oct 30 18:19:46.583319 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:46.583244 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:19:46.583799 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:46.583768 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:19:46.584555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:46.584535 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:19:46.622567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:46.622543 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj"] Oct 30 18:19:46.656685 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:46.656649 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7"] Oct 30 18:19:51.629779 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:51.625575 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj" podUID="70304573-b29c-45ef-933e-f974d4bac229" containerName="payload-processing" containerID="cri-o://41cb178b155bd3f77bcb0ad2ee393b95deb0d5b6317994fad0dd5bf0273d4cca" gracePeriod=25 Oct 30 18:19:51.693183 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:51.657869 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7" podUID="13bcd21f-ccd6-4d11-bb75-de03bc2683e6" containerName="payload-pre-processing" containerID="cri-o://b5285393ae8c28349174417e7c41d6c46a7c3ccb74b2afba8bd3927d7abb8a36" gracePeriod=25 Oct 30 18:19:51.972578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:51.972539 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g"] Oct 30 18:19:51.995950 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:51.990546 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-a90958fa-26q5g"] Oct 30 18:19:52.131646 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.112779 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj" Oct 30 18:19:52.220590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.216708 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2"] Oct 30 18:19:52.220590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.216988 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" podUID="19303a05-780a-4c03-9df6-2db3aeee3a9b" containerName="istio-proxy" containerID="cri-o://6a1a8c80a4f4e9ea5a7b6234181db2931b6b2977b6f5c5ec6a3a16faa63e7bca" gracePeriod=30 Oct 30 18:19:52.259355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.259331 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7" Oct 30 18:19:52.260037 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.260013 5232 generic.go:356] "Generic (PLEG): container finished" podID="70304573-b29c-45ef-933e-f974d4bac229" containerID="41cb178b155bd3f77bcb0ad2ee393b95deb0d5b6317994fad0dd5bf0273d4cca" exitCode=0 Oct 30 18:19:52.260176 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.260050 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj" Oct 30 18:19:52.260176 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.260090 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj" event={"ID":"70304573-b29c-45ef-933e-f974d4bac229","Type":"ContainerDied","Data":"41cb178b155bd3f77bcb0ad2ee393b95deb0d5b6317994fad0dd5bf0273d4cca"} Oct 30 18:19:52.260176 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.260147 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj" event={"ID":"70304573-b29c-45ef-933e-f974d4bac229","Type":"ContainerDied","Data":"2d1c24c0f67dcca47fdfe719742651add5b7a57991d4fe0783b4d160610a8ab0"} Oct 30 18:19:52.260339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.260179 5232 scope.go:122] "RemoveContainer" containerID="41cb178b155bd3f77bcb0ad2ee393b95deb0d5b6317994fad0dd5bf0273d4cca" Oct 30 18:19:52.261819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.261753 5232 generic.go:356] "Generic (PLEG): container finished" podID="13bcd21f-ccd6-4d11-bb75-de03bc2683e6" containerID="b5285393ae8c28349174417e7c41d6c46a7c3ccb74b2afba8bd3927d7abb8a36" exitCode=0 Oct 30 18:19:52.261819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.261789 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7" Oct 30 18:19:52.261819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.261813 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7" event={"ID":"13bcd21f-ccd6-4d11-bb75-de03bc2683e6","Type":"ContainerDied","Data":"b5285393ae8c28349174417e7c41d6c46a7c3ccb74b2afba8bd3927d7abb8a36"} Oct 30 18:19:52.262150 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.261853 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7" event={"ID":"13bcd21f-ccd6-4d11-bb75-de03bc2683e6","Type":"ContainerDied","Data":"c2234b290e4e7adda65dfd6d0099118a1dc237e7ed66a016815a4f62bc942c64"} Oct 30 18:19:52.271804 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.271789 5232 scope.go:122] "RemoveContainer" containerID="41cb178b155bd3f77bcb0ad2ee393b95deb0d5b6317994fad0dd5bf0273d4cca" Oct 30 18:19:52.272076 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:19:52.272052 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41cb178b155bd3f77bcb0ad2ee393b95deb0d5b6317994fad0dd5bf0273d4cca\": container with ID starting with 41cb178b155bd3f77bcb0ad2ee393b95deb0d5b6317994fad0dd5bf0273d4cca not found: ID does not exist" containerID="41cb178b155bd3f77bcb0ad2ee393b95deb0d5b6317994fad0dd5bf0273d4cca" Oct 30 18:19:52.272163 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.272088 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41cb178b155bd3f77bcb0ad2ee393b95deb0d5b6317994fad0dd5bf0273d4cca"} err="failed to get container status \"41cb178b155bd3f77bcb0ad2ee393b95deb0d5b6317994fad0dd5bf0273d4cca\": rpc error: code = NotFound desc = could not find container \"41cb178b155bd3f77bcb0ad2ee393b95deb0d5b6317994fad0dd5bf0273d4cca\": container with ID starting with 41cb178b155bd3f77bcb0ad2ee393b95deb0d5b6317994fad0dd5bf0273d4cca not found: ID does not exist" Oct 30 18:19:52.272163 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.272112 5232 scope.go:122] "RemoveContainer" containerID="b5285393ae8c28349174417e7c41d6c46a7c3ccb74b2afba8bd3927d7abb8a36" Oct 30 18:19:52.285403 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.285384 5232 scope.go:122] "RemoveContainer" containerID="b5285393ae8c28349174417e7c41d6c46a7c3ccb74b2afba8bd3927d7abb8a36" Oct 30 18:19:52.285728 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:19:52.285706 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5285393ae8c28349174417e7c41d6c46a7c3ccb74b2afba8bd3927d7abb8a36\": container with ID starting with b5285393ae8c28349174417e7c41d6c46a7c3ccb74b2afba8bd3927d7abb8a36 not found: ID does not exist" containerID="b5285393ae8c28349174417e7c41d6c46a7c3ccb74b2afba8bd3927d7abb8a36" Oct 30 18:19:52.285807 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.285738 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5285393ae8c28349174417e7c41d6c46a7c3ccb74b2afba8bd3927d7abb8a36"} err="failed to get container status \"b5285393ae8c28349174417e7c41d6c46a7c3ccb74b2afba8bd3927d7abb8a36\": rpc error: code = NotFound desc = could not find container \"b5285393ae8c28349174417e7c41d6c46a7c3ccb74b2afba8bd3927d7abb8a36\": container with ID starting with b5285393ae8c28349174417e7c41d6c46a7c3ccb74b2afba8bd3927d7abb8a36 not found: ID does not exist" Oct 30 18:19:52.328551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.316566 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/70304573-b29c-45ef-933e-f974d4bac229-kube-api-access-g6wgr\" (UniqueName: \"kubernetes.io/projected/70304573-b29c-45ef-933e-f974d4bac229-kube-api-access-g6wgr\") pod \"70304573-b29c-45ef-933e-f974d4bac229\" (UID: \"70304573-b29c-45ef-933e-f974d4bac229\") " Oct 30 18:19:52.328551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.316615 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/70304573-b29c-45ef-933e-f974d4bac229-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70304573-b29c-45ef-933e-f974d4bac229-plugins-config-volume\") pod \"70304573-b29c-45ef-933e-f974d4bac229\" (UID: \"70304573-b29c-45ef-933e-f974d4bac229\") " Oct 30 18:19:52.328551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.317757 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/70304573-b29c-45ef-933e-f974d4bac229-plugins-config-volume" pod "70304573-b29c-45ef-933e-f974d4bac229" (UID: "70304573-b29c-45ef-933e-f974d4bac229"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:52.328551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.321198 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70304573-b29c-45ef-933e-f974d4bac229-kube-api-access-g6wgr" pod "70304573-b29c-45ef-933e-f974d4bac229" (UID: "70304573-b29c-45ef-933e-f974d4bac229"). InnerVolumeSpecName "kube-api-access-g6wgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:52.430217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.423295 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13bcd21f-ccd6-4d11-bb75-de03bc2683e6-kube-api-access-n5f4d\" (UniqueName: \"kubernetes.io/projected/13bcd21f-ccd6-4d11-bb75-de03bc2683e6-kube-api-access-n5f4d\") pod \"13bcd21f-ccd6-4d11-bb75-de03bc2683e6\" (UID: \"13bcd21f-ccd6-4d11-bb75-de03bc2683e6\") " Oct 30 18:19:52.430217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.423361 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13bcd21f-ccd6-4d11-bb75-de03bc2683e6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13bcd21f-ccd6-4d11-bb75-de03bc2683e6-plugins-config-volume\") pod \"13bcd21f-ccd6-4d11-bb75-de03bc2683e6\" (UID: \"13bcd21f-ccd6-4d11-bb75-de03bc2683e6\") " Oct 30 18:19:52.430217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.424187 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6wgr\" (UniqueName: \"kubernetes.io/projected/70304573-b29c-45ef-933e-f974d4bac229-kube-api-access-g6wgr\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:52.430217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.424210 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/70304573-b29c-45ef-933e-f974d4bac229-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:52.469737 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.434063 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13bcd21f-ccd6-4d11-bb75-de03bc2683e6-plugins-config-volume" pod "13bcd21f-ccd6-4d11-bb75-de03bc2683e6" (UID: "13bcd21f-ccd6-4d11-bb75-de03bc2683e6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:52.469737 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.450622 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13bcd21f-ccd6-4d11-bb75-de03bc2683e6-kube-api-access-n5f4d" pod "13bcd21f-ccd6-4d11-bb75-de03bc2683e6" (UID: "13bcd21f-ccd6-4d11-bb75-de03bc2683e6"). InnerVolumeSpecName "kube-api-access-n5f4d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:52.562695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.562580 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n5f4d\" (UniqueName: \"kubernetes.io/projected/13bcd21f-ccd6-4d11-bb75-de03bc2683e6-kube-api-access-n5f4d\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:52.602782 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.602751 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/13bcd21f-ccd6-4d11-bb75-de03bc2683e6-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:52.762430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.762396 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj"] Oct 30 18:19:52.769110 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.769078 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-a90958fa-755c99bc46-bklhj"] Oct 30 18:19:52.805292 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.805263 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bbacde3-9498-4474-a6f1-7647c1392f9f" path="/var/lib/kubelet/pods/6bbacde3-9498-4474-a6f1-7647c1392f9f/volumes" Oct 30 18:19:52.805777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.805762 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="70304573-b29c-45ef-933e-f974d4bac229" path="/var/lib/kubelet/pods/70304573-b29c-45ef-933e-f974d4bac229/volumes" Oct 30 18:19:52.806087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.806073 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7"] Oct 30 18:19:52.806129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:52.806102 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-a90958fa-649c4cb67f-dsdn7"] Oct 30 18:19:54.815286 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:54.812270 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13bcd21f-ccd6-4d11-bb75-de03bc2683e6" path="/var/lib/kubelet/pods/13bcd21f-ccd6-4d11-bb75-de03bc2683e6/volumes" Oct 30 18:19:55.839375 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:55.839345 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59"] Oct 30 18:19:55.840203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:55.840182 5232 cpu_manager.go:414] "containerMap: removing container" podUID="13bcd21f-ccd6-4d11-bb75-de03bc2683e6" containerName="payload-pre-processing" Oct 30 18:19:55.840271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:55.840205 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13bcd21f-ccd6-4d11-bb75-de03bc2683e6" containerName="payload-pre-processing" Oct 30 18:19:55.840271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:55.840215 5232 cpu_manager.go:414] "containerMap: removing container" podUID="6bbacde3-9498-4474-a6f1-7647c1392f9f" containerName="revoke-keys" Oct 30 18:19:55.840271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:55.840220 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bbacde3-9498-4474-a6f1-7647c1392f9f" containerName="revoke-keys" Oct 30 18:19:55.840271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:55.840246 5232 cpu_manager.go:414] "containerMap: removing container" podUID="70304573-b29c-45ef-933e-f974d4bac229" containerName="payload-processing" Oct 30 18:19:55.840271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:55.840251 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="70304573-b29c-45ef-933e-f974d4bac229" containerName="payload-processing" Oct 30 18:19:55.840428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:55.840365 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="70304573-b29c-45ef-933e-f974d4bac229" containerName="payload-processing" Oct 30 18:19:55.840428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:55.840380 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="13bcd21f-ccd6-4d11-bb75-de03bc2683e6" containerName="payload-pre-processing" Oct 30 18:19:55.840428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:55.840386 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bbacde3-9498-4474-a6f1-7647c1392f9f" containerName="revoke-keys" Oct 30 18:19:55.845848 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:55.845827 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59" Oct 30 18:19:55.862059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:55.862031 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59"] Oct 30 18:19:55.867366 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:55.867341 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:19:55.867476 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:55.867422 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:19:55.936016 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:55.935985 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d20871-227a-4ba9-82a7-ae48efbe014b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0093b00c-2xj59\" (UID: \"c2d20871-227a-4ba9-82a7-ae48efbe014b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59" Oct 30 18:19:55.936195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:55.936042 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d20871-227a-4ba9-82a7-ae48efbe014b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0093b00c-2xj59\" (UID: \"c2d20871-227a-4ba9-82a7-ae48efbe014b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59" Oct 30 18:19:55.936195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:55.936113 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fbfk\" (UniqueName: \"kubernetes.io/projected/c2d20871-227a-4ba9-82a7-ae48efbe014b-kube-api-access-2fbfk\") pod \"maas-api-revoke-sub-0093b00c-2xj59\" (UID: \"c2d20871-227a-4ba9-82a7-ae48efbe014b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59" Oct 30 18:19:56.052930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:56.043974 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d20871-227a-4ba9-82a7-ae48efbe014b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0093b00c-2xj59\" (UID: \"c2d20871-227a-4ba9-82a7-ae48efbe014b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59" Oct 30 18:19:56.053296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:56.053269 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d20871-227a-4ba9-82a7-ae48efbe014b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0093b00c-2xj59\" (UID: \"c2d20871-227a-4ba9-82a7-ae48efbe014b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59" Oct 30 18:19:56.053999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:56.053970 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fbfk\" (UniqueName: \"kubernetes.io/projected/c2d20871-227a-4ba9-82a7-ae48efbe014b-kube-api-access-2fbfk\") pod \"maas-api-revoke-sub-0093b00c-2xj59\" (UID: \"c2d20871-227a-4ba9-82a7-ae48efbe014b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59" Oct 30 18:19:56.055025 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:56.049292 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d20871-227a-4ba9-82a7-ae48efbe014b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0093b00c-2xj59\" (UID: \"c2d20871-227a-4ba9-82a7-ae48efbe014b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59" Oct 30 18:19:56.106612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:56.084942 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d20871-227a-4ba9-82a7-ae48efbe014b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0093b00c-2xj59\" (UID: \"c2d20871-227a-4ba9-82a7-ae48efbe014b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59" Oct 30 18:19:56.106612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:56.094745 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fbfk\" (UniqueName: \"kubernetes.io/projected/c2d20871-227a-4ba9-82a7-ae48efbe014b-kube-api-access-2fbfk\") pod \"maas-api-revoke-sub-0093b00c-2xj59\" (UID: \"c2d20871-227a-4ba9-82a7-ae48efbe014b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59" Oct 30 18:19:56.162994 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:56.162962 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59" Oct 30 18:19:56.478016 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:56.477941 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59"] Oct 30 18:19:56.482781 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:19:56.482751 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d20871_227a_4ba9_82a7_ae48efbe014b.slice/crio-dc405c5987d2b43d9f65b47c06b9a62502d1e1213e870841e3a2ae74b97ff006 WatchSource:0}: Error finding container dc405c5987d2b43d9f65b47c06b9a62502d1e1213e870841e3a2ae74b97ff006: Status 404 returned error can't find the container with id dc405c5987d2b43d9f65b47c06b9a62502d1e1213e870841e3a2ae74b97ff006 Oct 30 18:19:56.604156 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:56.604119 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59" event={"ID":"c2d20871-227a-4ba9-82a7-ae48efbe014b","Type":"ContainerStarted","Data":"dc405c5987d2b43d9f65b47c06b9a62502d1e1213e870841e3a2ae74b97ff006"} Oct 30 18:19:57.034355 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:19:57.034315 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:19:57.678751 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.678712 5232 generic.go:356] "Generic (PLEG): container finished" podID="c2d20871-227a-4ba9-82a7-ae48efbe014b" containerID="83f572f64a38a5d19b391a47d8eb61e8a38202a8df7ea0e641cfd804008c5747" exitCode=0 Oct 30 18:19:57.678938 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.678794 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59" event={"ID":"c2d20871-227a-4ba9-82a7-ae48efbe014b","Type":"ContainerDied","Data":"83f572f64a38a5d19b391a47d8eb61e8a38202a8df7ea0e641cfd804008c5747"} Oct 30 18:19:57.681784 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.681760 5232 generic.go:356] "Generic (PLEG): container finished" podID="19303a05-780a-4c03-9df6-2db3aeee3a9b" containerID="6a1a8c80a4f4e9ea5a7b6234181db2931b6b2977b6f5c5ec6a3a16faa63e7bca" exitCode=0 Oct 30 18:19:57.681942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.681837 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" event={"ID":"19303a05-780a-4c03-9df6-2db3aeee3a9b","Type":"ContainerDied","Data":"6a1a8c80a4f4e9ea5a7b6234181db2931b6b2977b6f5c5ec6a3a16faa63e7bca"} Oct 30 18:19:57.681942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.681890 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" event={"ID":"19303a05-780a-4c03-9df6-2db3aeee3a9b","Type":"ContainerDied","Data":"27dce2ba098603e59827c12a839424b209eb66b245c3c06b68e0bbf6d6cbefab"} Oct 30 18:19:57.681942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.681910 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="27dce2ba098603e59827c12a839424b209eb66b245c3c06b68e0bbf6d6cbefab" Oct 30 18:19:57.681942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.681923 5232 scope.go:122] "RemoveContainer" containerID="437a7ca20c9e513770e6ef040e776c44c14c94542879f7b5613757bb47bbcdcf" Oct 30 18:19:57.713080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.713061 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:19:57.807137 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.807097 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-podinfo\") pod \"19303a05-780a-4c03-9df6-2db3aeee3a9b\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " Oct 30 18:19:57.807137 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.807156 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-token\" (UniqueName: \"kubernetes.io/projected/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-token\") pod \"19303a05-780a-4c03-9df6-2db3aeee3a9b\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " Oct 30 18:19:57.807403 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.807250 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-envoy\") pod \"19303a05-780a-4c03-9df6-2db3aeee3a9b\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " Oct 30 18:19:57.807403 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.807290 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-data\") pod \"19303a05-780a-4c03-9df6-2db3aeee3a9b\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " Oct 30 18:19:57.807403 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.807399 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-workload-certs\") pod \"19303a05-780a-4c03-9df6-2db3aeee3a9b\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " Oct 30 18:19:57.807601 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.807437 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-workload-socket\") pod \"19303a05-780a-4c03-9df6-2db3aeee3a9b\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " Oct 30 18:19:57.807601 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.807535 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/19303a05-780a-4c03-9df6-2db3aeee3a9b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19303a05-780a-4c03-9df6-2db3aeee3a9b-istiod-ca-cert\") pod \"19303a05-780a-4c03-9df6-2db3aeee3a9b\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " Oct 30 18:19:57.807601 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.807585 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/19303a05-780a-4c03-9df6-2db3aeee3a9b-kube-api-access-jjhmj\" (UniqueName: \"kubernetes.io/projected/19303a05-780a-4c03-9df6-2db3aeee3a9b-kube-api-access-jjhmj\") pod \"19303a05-780a-4c03-9df6-2db3aeee3a9b\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " Oct 30 18:19:57.807768 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.807660 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-credential-socket\") pod \"19303a05-780a-4c03-9df6-2db3aeee3a9b\" (UID: \"19303a05-780a-4c03-9df6-2db3aeee3a9b\") " Oct 30 18:19:57.807768 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.807734 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-data" pod "19303a05-780a-4c03-9df6-2db3aeee3a9b" (UID: "19303a05-780a-4c03-9df6-2db3aeee3a9b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:57.807883 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.807812 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-workload-socket" pod "19303a05-780a-4c03-9df6-2db3aeee3a9b" (UID: "19303a05-780a-4c03-9df6-2db3aeee3a9b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:57.808109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.808085 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-credential-socket" pod "19303a05-780a-4c03-9df6-2db3aeee3a9b" (UID: "19303a05-780a-4c03-9df6-2db3aeee3a9b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:57.808213 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.808182 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/19303a05-780a-4c03-9df6-2db3aeee3a9b-istiod-ca-cert" pod "19303a05-780a-4c03-9df6-2db3aeee3a9b" (UID: "19303a05-780a-4c03-9df6-2db3aeee3a9b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:57.808532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.808480 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:57.808710 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.808691 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:57.808827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.808813 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/19303a05-780a-4c03-9df6-2db3aeee3a9b-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:57.808925 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.808912 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:57.809066 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.809045 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-workload-certs" pod "19303a05-780a-4c03-9df6-2db3aeee3a9b" (UID: "19303a05-780a-4c03-9df6-2db3aeee3a9b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:57.810148 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.810121 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-podinfo" pod "19303a05-780a-4c03-9df6-2db3aeee3a9b" (UID: "19303a05-780a-4c03-9df6-2db3aeee3a9b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:19:57.810292 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.810275 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19303a05-780a-4c03-9df6-2db3aeee3a9b-kube-api-access-jjhmj" pod "19303a05-780a-4c03-9df6-2db3aeee3a9b" (UID: "19303a05-780a-4c03-9df6-2db3aeee3a9b"). InnerVolumeSpecName "kube-api-access-jjhmj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:57.810834 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.810800 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-envoy" pod "19303a05-780a-4c03-9df6-2db3aeee3a9b" (UID: "19303a05-780a-4c03-9df6-2db3aeee3a9b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:19:57.810942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.810920 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-token" pod "19303a05-780a-4c03-9df6-2db3aeee3a9b" (UID: "19303a05-780a-4c03-9df6-2db3aeee3a9b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:57.910492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.910428 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:57.910492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.910454 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jjhmj\" (UniqueName: \"kubernetes.io/projected/19303a05-780a-4c03-9df6-2db3aeee3a9b-kube-api-access-jjhmj\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:57.910492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.910464 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:57.910492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.910474 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:57.910492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:57.910483 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/19303a05-780a-4c03-9df6-2db3aeee3a9b-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:58.693560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:58.693487 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2" Oct 30 18:19:58.807670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:58.807639 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2"] Oct 30 18:19:58.809677 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:58.809651 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-a90958fa-openshift-default-844f8b6d7c-6s2d2"] Oct 30 18:19:58.929538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:58.914098 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59" Oct 30 18:19:58.929538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:58.929410 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c2d20871-227a-4ba9-82a7-ae48efbe014b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d20871-227a-4ba9-82a7-ae48efbe014b-maas-api-xks-ca\") pod \"c2d20871-227a-4ba9-82a7-ae48efbe014b\" (UID: \"c2d20871-227a-4ba9-82a7-ae48efbe014b\") " Oct 30 18:19:58.947582 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:58.943322 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c2d20871-227a-4ba9-82a7-ae48efbe014b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d20871-227a-4ba9-82a7-ae48efbe014b-maas-api-openshift-ca\") pod \"c2d20871-227a-4ba9-82a7-ae48efbe014b\" (UID: \"c2d20871-227a-4ba9-82a7-ae48efbe014b\") " Oct 30 18:19:58.947582 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:58.943436 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c2d20871-227a-4ba9-82a7-ae48efbe014b-kube-api-access-2fbfk\" (UniqueName: \"kubernetes.io/projected/c2d20871-227a-4ba9-82a7-ae48efbe014b-kube-api-access-2fbfk\") pod \"c2d20871-227a-4ba9-82a7-ae48efbe014b\" (UID: \"c2d20871-227a-4ba9-82a7-ae48efbe014b\") " Oct 30 18:19:58.969535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:58.949026 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2d20871-227a-4ba9-82a7-ae48efbe014b-maas-api-openshift-ca" pod "c2d20871-227a-4ba9-82a7-ae48efbe014b" (UID: "c2d20871-227a-4ba9-82a7-ae48efbe014b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:19:59.009537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:58.992940 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2d20871-227a-4ba9-82a7-ae48efbe014b-kube-api-access-2fbfk" pod "c2d20871-227a-4ba9-82a7-ae48efbe014b" (UID: "c2d20871-227a-4ba9-82a7-ae48efbe014b"). InnerVolumeSpecName "kube-api-access-2fbfk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:19:59.084538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:59.072828 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c2d20871-227a-4ba9-82a7-ae48efbe014b-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:59.084538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:59.072866 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2fbfk\" (UniqueName: \"kubernetes.io/projected/c2d20871-227a-4ba9-82a7-ae48efbe014b-kube-api-access-2fbfk\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:59.097746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:59.097711 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c2d20871-227a-4ba9-82a7-ae48efbe014b-maas-api-xks-ca" pod "c2d20871-227a-4ba9-82a7-ae48efbe014b" (UID: "c2d20871-227a-4ba9-82a7-ae48efbe014b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:19:59.191532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:59.177123 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c2d20871-227a-4ba9-82a7-ae48efbe014b-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:19:59.703949 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:59.703918 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59" event={"ID":"c2d20871-227a-4ba9-82a7-ae48efbe014b","Type":"ContainerDied","Data":"dc405c5987d2b43d9f65b47c06b9a62502d1e1213e870841e3a2ae74b97ff006"} Oct 30 18:19:59.703949 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:59.703935 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59" Oct 30 18:19:59.704372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:19:59.703965 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dc405c5987d2b43d9f65b47c06b9a62502d1e1213e870841e3a2ae74b97ff006" Oct 30 18:20:00.804903 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:00.804870 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="19303a05-780a-4c03-9df6-2db3aeee3a9b" path="/var/lib/kubelet/pods/19303a05-780a-4c03-9df6-2db3aeee3a9b/volumes" Oct 30 18:20:01.833350 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:01.833316 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t"] Oct 30 18:20:01.834047 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:01.834031 5232 cpu_manager.go:414] "containerMap: removing container" podUID="19303a05-780a-4c03-9df6-2db3aeee3a9b" containerName="istio-proxy" Oct 30 18:20:01.834101 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:01.834048 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19303a05-780a-4c03-9df6-2db3aeee3a9b" containerName="istio-proxy" Oct 30 18:20:01.834101 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:01.834065 5232 cpu_manager.go:414] "containerMap: removing container" podUID="c2d20871-227a-4ba9-82a7-ae48efbe014b" containerName="revoke-keys" Oct 30 18:20:01.834101 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:01.834071 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c2d20871-227a-4ba9-82a7-ae48efbe014b" containerName="revoke-keys" Oct 30 18:20:01.834101 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:01.834100 5232 cpu_manager.go:414] "containerMap: removing container" podUID="19303a05-780a-4c03-9df6-2db3aeee3a9b" containerName="istio-proxy" Oct 30 18:20:01.834225 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:01.834107 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="19303a05-780a-4c03-9df6-2db3aeee3a9b" containerName="istio-proxy" Oct 30 18:20:01.834225 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:01.834202 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="19303a05-780a-4c03-9df6-2db3aeee3a9b" containerName="istio-proxy" Oct 30 18:20:01.834225 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:01.834210 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="c2d20871-227a-4ba9-82a7-ae48efbe014b" containerName="revoke-keys" Oct 30 18:20:01.834225 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:01.834218 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="19303a05-780a-4c03-9df6-2db3aeee3a9b" containerName="istio-proxy" Oct 30 18:20:01.838224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:01.838204 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t" Oct 30 18:20:01.854982 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:01.854958 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t"] Oct 30 18:20:01.861106 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:01.861081 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:20:01.861262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:01.861120 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:20:01.942008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:01.941976 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b914d5ad-67c2-4e60-b334-3d5b18dbe897-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0171017-d8t5t\" (UID: \"b914d5ad-67c2-4e60-b334-3d5b18dbe897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t" Oct 30 18:20:01.942008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:01.942014 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lrxgl\" (UniqueName: \"kubernetes.io/projected/b914d5ad-67c2-4e60-b334-3d5b18dbe897-kube-api-access-lrxgl\") pod \"maas-api-revoke-keys-e2e-ait-b0171017-d8t5t\" (UID: \"b914d5ad-67c2-4e60-b334-3d5b18dbe897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t" Oct 30 18:20:01.942262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:01.942240 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b914d5ad-67c2-4e60-b334-3d5b18dbe897-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0171017-d8t5t\" (UID: \"b914d5ad-67c2-4e60-b334-3d5b18dbe897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t" Oct 30 18:20:02.043853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:02.043811 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b914d5ad-67c2-4e60-b334-3d5b18dbe897-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0171017-d8t5t\" (UID: \"b914d5ad-67c2-4e60-b334-3d5b18dbe897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t" Oct 30 18:20:02.043853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:02.043858 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lrxgl\" (UniqueName: \"kubernetes.io/projected/b914d5ad-67c2-4e60-b334-3d5b18dbe897-kube-api-access-lrxgl\") pod \"maas-api-revoke-keys-e2e-ait-b0171017-d8t5t\" (UID: \"b914d5ad-67c2-4e60-b334-3d5b18dbe897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t" Oct 30 18:20:02.044102 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:02.044009 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b914d5ad-67c2-4e60-b334-3d5b18dbe897-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0171017-d8t5t\" (UID: \"b914d5ad-67c2-4e60-b334-3d5b18dbe897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t" Oct 30 18:20:02.044413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:02.044393 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b914d5ad-67c2-4e60-b334-3d5b18dbe897-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0171017-d8t5t\" (UID: \"b914d5ad-67c2-4e60-b334-3d5b18dbe897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t" Oct 30 18:20:02.046549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:02.046503 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b914d5ad-67c2-4e60-b334-3d5b18dbe897-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-b0171017-d8t5t\" (UID: \"b914d5ad-67c2-4e60-b334-3d5b18dbe897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t" Oct 30 18:20:02.063002 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:02.062983 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lrxgl\" (UniqueName: \"kubernetes.io/projected/b914d5ad-67c2-4e60-b334-3d5b18dbe897-kube-api-access-lrxgl\") pod \"maas-api-revoke-keys-e2e-ait-b0171017-d8t5t\" (UID: \"b914d5ad-67c2-4e60-b334-3d5b18dbe897\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t" Oct 30 18:20:02.155573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:02.155486 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t" Oct 30 18:20:02.463075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:02.463044 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t"] Oct 30 18:20:02.463657 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:20:02.463631 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb914d5ad_67c2_4e60_b334_3d5b18dbe897.slice/crio-f70358ffb86c3a9ebdd9cf8c9f0c9f2d93cd24aa9e4e23bae81c1cb30a71147a WatchSource:0}: Error finding container f70358ffb86c3a9ebdd9cf8c9f0c9f2d93cd24aa9e4e23bae81c1cb30a71147a: Status 404 returned error can't find the container with id f70358ffb86c3a9ebdd9cf8c9f0c9f2d93cd24aa9e4e23bae81c1cb30a71147a Oct 30 18:20:02.741593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:02.741554 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t" event={"ID":"b914d5ad-67c2-4e60-b334-3d5b18dbe897","Type":"ContainerStarted","Data":"f70358ffb86c3a9ebdd9cf8c9f0c9f2d93cd24aa9e4e23bae81c1cb30a71147a"} Oct 30 18:20:03.752393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:03.752354 5232 generic.go:356] "Generic (PLEG): container finished" podID="b914d5ad-67c2-4e60-b334-3d5b18dbe897" containerID="2e57e08fb7173869653c5c5052b9592bdd179ec9b6aa7be040cf433bfd2dedba" exitCode=0 Oct 30 18:20:03.752883 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:03.752405 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t" event={"ID":"b914d5ad-67c2-4e60-b334-3d5b18dbe897","Type":"ContainerDied","Data":"2e57e08fb7173869653c5c5052b9592bdd179ec9b6aa7be040cf433bfd2dedba"} Oct 30 18:20:04.928320 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:04.928293 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t" Oct 30 18:20:05.012886 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:05.012803 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b914d5ad-67c2-4e60-b334-3d5b18dbe897-kube-api-access-lrxgl\" (UniqueName: \"kubernetes.io/projected/b914d5ad-67c2-4e60-b334-3d5b18dbe897-kube-api-access-lrxgl\") pod \"b914d5ad-67c2-4e60-b334-3d5b18dbe897\" (UID: \"b914d5ad-67c2-4e60-b334-3d5b18dbe897\") " Oct 30 18:20:05.013085 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:05.013067 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b914d5ad-67c2-4e60-b334-3d5b18dbe897-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b914d5ad-67c2-4e60-b334-3d5b18dbe897-maas-api-openshift-ca\") pod \"b914d5ad-67c2-4e60-b334-3d5b18dbe897\" (UID: \"b914d5ad-67c2-4e60-b334-3d5b18dbe897\") " Oct 30 18:20:05.013156 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:05.013138 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b914d5ad-67c2-4e60-b334-3d5b18dbe897-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b914d5ad-67c2-4e60-b334-3d5b18dbe897-maas-api-xks-ca\") pod \"b914d5ad-67c2-4e60-b334-3d5b18dbe897\" (UID: \"b914d5ad-67c2-4e60-b334-3d5b18dbe897\") " Oct 30 18:20:05.013465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:05.013436 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b914d5ad-67c2-4e60-b334-3d5b18dbe897-maas-api-openshift-ca" pod "b914d5ad-67c2-4e60-b334-3d5b18dbe897" (UID: "b914d5ad-67c2-4e60-b334-3d5b18dbe897"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:05.013743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:05.013714 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b914d5ad-67c2-4e60-b334-3d5b18dbe897-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:20:05.015233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:05.015196 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b914d5ad-67c2-4e60-b334-3d5b18dbe897-kube-api-access-lrxgl" pod "b914d5ad-67c2-4e60-b334-3d5b18dbe897" (UID: "b914d5ad-67c2-4e60-b334-3d5b18dbe897"). InnerVolumeSpecName "kube-api-access-lrxgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:05.015431 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:05.015406 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b914d5ad-67c2-4e60-b334-3d5b18dbe897-maas-api-xks-ca" pod "b914d5ad-67c2-4e60-b334-3d5b18dbe897" (UID: "b914d5ad-67c2-4e60-b334-3d5b18dbe897"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:20:05.115179 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:05.115149 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b914d5ad-67c2-4e60-b334-3d5b18dbe897-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:20:05.115179 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:05.115177 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lrxgl\" (UniqueName: \"kubernetes.io/projected/b914d5ad-67c2-4e60-b334-3d5b18dbe897-kube-api-access-lrxgl\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:20:05.771495 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:05.771454 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t" event={"ID":"b914d5ad-67c2-4e60-b334-3d5b18dbe897","Type":"ContainerDied","Data":"f70358ffb86c3a9ebdd9cf8c9f0c9f2d93cd24aa9e4e23bae81c1cb30a71147a"} Oct 30 18:20:05.771495 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:05.771484 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t" Oct 30 18:20:05.771756 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:05.771580 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f70358ffb86c3a9ebdd9cf8c9f0c9f2d93cd24aa9e4e23bae81c1cb30a71147a" Oct 30 18:20:14.264408 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:14.264376 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr"] Oct 30 18:20:14.297868 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:14.297837 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp"] Oct 30 18:20:18.368198 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:18.368165 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t"] Oct 30 18:20:18.380157 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:18.380126 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-b0171017-d8t5t"] Oct 30 18:20:18.807036 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:18.807006 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b914d5ad-67c2-4e60-b334-3d5b18dbe897" path="/var/lib/kubelet/pods/b914d5ad-67c2-4e60-b334-3d5b18dbe897/volumes" Oct 30 18:20:19.284535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:19.264994 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr" podUID="1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77" containerName="payload-processing" containerID="cri-o://c95c0a7694561d638bf94aec8bc9367fd8a39bce884c660c9f13ce2c8f575f57" gracePeriod=25 Oct 30 18:20:19.304301 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:19.299617 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" podUID="430d86ee-8426-4d48-9441-19e1366fc46e" containerName="payload-pre-processing" containerID="cri-o://21a7ffb6e8d75434e1cae83ebf66568509853cfe16e9ab7bff0faca55176b000" gracePeriod=25 Oct 30 18:20:19.681326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:19.681299 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" Oct 30 18:20:19.706386 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:19.706356 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr" Oct 30 18:20:19.775342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:19.775304 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/430d86ee-8426-4d48-9441-19e1366fc46e-kube-api-access-jsgh2\" (UniqueName: \"kubernetes.io/projected/430d86ee-8426-4d48-9441-19e1366fc46e-kube-api-access-jsgh2\") pod \"430d86ee-8426-4d48-9441-19e1366fc46e\" (UID: \"430d86ee-8426-4d48-9441-19e1366fc46e\") " Oct 30 18:20:19.775613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:19.775592 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77-kube-api-access-z46h6\" (UniqueName: \"kubernetes.io/projected/1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77-kube-api-access-z46h6\") pod \"1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77\" (UID: \"1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77\") " Oct 30 18:20:19.775749 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:19.775729 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/430d86ee-8426-4d48-9441-19e1366fc46e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430d86ee-8426-4d48-9441-19e1366fc46e-plugins-config-volume\") pod \"430d86ee-8426-4d48-9441-19e1366fc46e\" (UID: \"430d86ee-8426-4d48-9441-19e1366fc46e\") " Oct 30 18:20:19.775995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:19.775974 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77-plugins-config-volume\") pod \"1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77\" (UID: \"1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77\") " Oct 30 18:20:19.776162 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:19.776131 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/430d86ee-8426-4d48-9441-19e1366fc46e-plugins-config-volume" pod "430d86ee-8426-4d48-9441-19e1366fc46e" (UID: "430d86ee-8426-4d48-9441-19e1366fc46e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:19.776405 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:19.776386 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77-plugins-config-volume" pod "1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77" (UID: "1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:20:19.777162 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:19.777132 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/430d86ee-8426-4d48-9441-19e1366fc46e-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:20:19.777279 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:19.777164 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:20:19.777972 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:19.777927 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77-kube-api-access-z46h6" pod "1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77" (UID: "1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77"). InnerVolumeSpecName "kube-api-access-z46h6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:19.778246 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:19.778226 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/430d86ee-8426-4d48-9441-19e1366fc46e-kube-api-access-jsgh2" pod "430d86ee-8426-4d48-9441-19e1366fc46e" (UID: "430d86ee-8426-4d48-9441-19e1366fc46e"). InnerVolumeSpecName "kube-api-access-jsgh2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:20:19.878366 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:19.878282 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jsgh2\" (UniqueName: \"kubernetes.io/projected/430d86ee-8426-4d48-9441-19e1366fc46e-kube-api-access-jsgh2\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:20:19.878366 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:19.878313 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z46h6\" (UniqueName: \"kubernetes.io/projected/1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77-kube-api-access-z46h6\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:20:20.167060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.166970 5232 generic.go:356] "Generic (PLEG): container finished" podID="1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77" containerID="c95c0a7694561d638bf94aec8bc9367fd8a39bce884c660c9f13ce2c8f575f57" exitCode=0 Oct 30 18:20:20.167060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.167021 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr" Oct 30 18:20:20.167260 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.167060 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr" event={"ID":"1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77","Type":"ContainerDied","Data":"c95c0a7694561d638bf94aec8bc9367fd8a39bce884c660c9f13ce2c8f575f57"} Oct 30 18:20:20.167260 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.167117 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr" event={"ID":"1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77","Type":"ContainerDied","Data":"b1fac97be40a7c4ace47bad5d40a3d0df0206fe8559ffa5a4359c6efcc5590a9"} Oct 30 18:20:20.167260 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.167146 5232 scope.go:122] "RemoveContainer" containerID="c95c0a7694561d638bf94aec8bc9367fd8a39bce884c660c9f13ce2c8f575f57" Oct 30 18:20:20.169141 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.169119 5232 generic.go:356] "Generic (PLEG): container finished" podID="430d86ee-8426-4d48-9441-19e1366fc46e" containerID="21a7ffb6e8d75434e1cae83ebf66568509853cfe16e9ab7bff0faca55176b000" exitCode=0 Oct 30 18:20:20.169277 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.169178 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" Oct 30 18:20:20.169277 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.169192 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" event={"ID":"430d86ee-8426-4d48-9441-19e1366fc46e","Type":"ContainerDied","Data":"21a7ffb6e8d75434e1cae83ebf66568509853cfe16e9ab7bff0faca55176b000"} Oct 30 18:20:20.169277 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.169234 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" event={"ID":"430d86ee-8426-4d48-9441-19e1366fc46e","Type":"ContainerDied","Data":"63954a13ea66d517a68c8f4a7c0c8389d92bf6a8a70c10782ae6294879fbb834"} Oct 30 18:20:20.181700 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.181678 5232 scope.go:122] "RemoveContainer" containerID="c95c0a7694561d638bf94aec8bc9367fd8a39bce884c660c9f13ce2c8f575f57" Oct 30 18:20:20.182056 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:20:20.182029 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c95c0a7694561d638bf94aec8bc9367fd8a39bce884c660c9f13ce2c8f575f57\": container with ID starting with c95c0a7694561d638bf94aec8bc9367fd8a39bce884c660c9f13ce2c8f575f57 not found: ID does not exist" containerID="c95c0a7694561d638bf94aec8bc9367fd8a39bce884c660c9f13ce2c8f575f57" Oct 30 18:20:20.182144 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.182069 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c95c0a7694561d638bf94aec8bc9367fd8a39bce884c660c9f13ce2c8f575f57"} err="failed to get container status \"c95c0a7694561d638bf94aec8bc9367fd8a39bce884c660c9f13ce2c8f575f57\": rpc error: code = NotFound desc = could not find container \"c95c0a7694561d638bf94aec8bc9367fd8a39bce884c660c9f13ce2c8f575f57\": container with ID starting with c95c0a7694561d638bf94aec8bc9367fd8a39bce884c660c9f13ce2c8f575f57 not found: ID does not exist" Oct 30 18:20:20.182144 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.182092 5232 scope.go:122] "RemoveContainer" containerID="21a7ffb6e8d75434e1cae83ebf66568509853cfe16e9ab7bff0faca55176b000" Oct 30 18:20:20.193582 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.193561 5232 scope.go:122] "RemoveContainer" containerID="21a7ffb6e8d75434e1cae83ebf66568509853cfe16e9ab7bff0faca55176b000" Oct 30 18:20:20.193921 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:20:20.193899 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"21a7ffb6e8d75434e1cae83ebf66568509853cfe16e9ab7bff0faca55176b000\": container with ID starting with 21a7ffb6e8d75434e1cae83ebf66568509853cfe16e9ab7bff0faca55176b000 not found: ID does not exist" containerID="21a7ffb6e8d75434e1cae83ebf66568509853cfe16e9ab7bff0faca55176b000" Oct 30 18:20:20.194000 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.193928 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"21a7ffb6e8d75434e1cae83ebf66568509853cfe16e9ab7bff0faca55176b000"} err="failed to get container status \"21a7ffb6e8d75434e1cae83ebf66568509853cfe16e9ab7bff0faca55176b000\": rpc error: code = NotFound desc = could not find container \"21a7ffb6e8d75434e1cae83ebf66568509853cfe16e9ab7bff0faca55176b000\": container with ID starting with 21a7ffb6e8d75434e1cae83ebf66568509853cfe16e9ab7bff0faca55176b000 not found: ID does not exist" Oct 30 18:20:20.233126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.233093 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr"] Oct 30 18:20:20.239178 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.239148 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-b0171017-8fc874664-qpttr"] Oct 30 18:20:20.273084 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.273054 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp"] Oct 30 18:20:20.278115 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.278089 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp"] Oct 30 18:20:20.591547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.584315 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ait-b0171017-695499df47-q4jkp" podUID="430d86ee-8426-4d48-9441-19e1366fc46e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.142:50052\" within 1s: context deadline exceeded" Oct 30 18:20:20.591547 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:20:20.584760 5232 logging.go:55] [core] [Channel #2039 SubChannel #2040]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.142:50052", ServerName: "10.133.0.142:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.142:50052: operation was canceled" Oct 30 18:20:20.888545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.888435 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77" path="/var/lib/kubelet/pods/1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77/volumes" Oct 30 18:20:20.889753 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:20.889729 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="430d86ee-8426-4d48-9441-19e1366fc46e" path="/var/lib/kubelet/pods/430d86ee-8426-4d48-9441-19e1366fc46e/volumes" Oct 30 18:20:26.906976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:26.906939 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w"] Oct 30 18:20:26.907834 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:26.907814 5232 cpu_manager.go:414] "containerMap: removing container" podUID="1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77" containerName="payload-processing" Oct 30 18:20:26.907907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:26.907836 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77" containerName="payload-processing" Oct 30 18:20:26.907907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:26.907881 5232 cpu_manager.go:414] "containerMap: removing container" podUID="b914d5ad-67c2-4e60-b334-3d5b18dbe897" containerName="revoke-keys" Oct 30 18:20:26.907907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:26.907886 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b914d5ad-67c2-4e60-b334-3d5b18dbe897" containerName="revoke-keys" Oct 30 18:20:26.907907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:26.907902 5232 cpu_manager.go:414] "containerMap: removing container" podUID="430d86ee-8426-4d48-9441-19e1366fc46e" containerName="payload-pre-processing" Oct 30 18:20:26.908078 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:26.907908 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="430d86ee-8426-4d48-9441-19e1366fc46e" containerName="payload-pre-processing" Oct 30 18:20:26.908078 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:26.908000 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="430d86ee-8426-4d48-9441-19e1366fc46e" containerName="payload-pre-processing" Oct 30 18:20:26.908078 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:26.908009 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c0c8f9b-05b3-4ce3-bef1-2518d99a8c77" containerName="payload-processing" Oct 30 18:20:26.908078 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:26.908022 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="b914d5ad-67c2-4e60-b334-3d5b18dbe897" containerName="revoke-keys" Oct 30 18:20:26.911760 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:26.911740 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:26.929556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:26.929523 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-b2b596-openshift-default-dockercfg-x4hhc\"" Oct 30 18:20:26.949283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:26.949256 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w"] Oct 30 18:20:27.000365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.000334 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-data\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.000569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.000374 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2vnd\" (UniqueName: \"kubernetes.io/projected/10b2ac99-f96c-4585-81cb-c28db96b7eb6-kube-api-access-l2vnd\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.000684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.000667 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-token\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.000745 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.000715 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-workload-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.000875 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.000850 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-envoy\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.000981 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.000964 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istiod-ca-cert\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.001040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.001023 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-credential-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.001169 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.001153 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-podinfo\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.001234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.001195 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-workload-certs\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.050841 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:20:27.050803 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:20:27.103082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.103047 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-token\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.103571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.103099 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-workload-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.103571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.103155 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-envoy\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.103571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.103209 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istiod-ca-cert\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.103571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.103250 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-credential-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.103571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.103349 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-podinfo\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.103571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.103388 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-workload-certs\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.103571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.103432 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-data\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.103571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.103456 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2vnd\" (UniqueName: \"kubernetes.io/projected/10b2ac99-f96c-4585-81cb-c28db96b7eb6-kube-api-access-l2vnd\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.104492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.104463 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-workload-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.105757 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.105730 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istiod-ca-cert\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.106297 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.105975 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-credential-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.106297 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.106196 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-workload-certs\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.106297 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.106258 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-data\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.107362 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.107339 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-envoy\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.107777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.107753 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-podinfo\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.122842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.122815 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2vnd\" (UniqueName: \"kubernetes.io/projected/10b2ac99-f96c-4585-81cb-c28db96b7eb6-kube-api-access-l2vnd\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.122842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.122829 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-token\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.185913 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.185849 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn"] Oct 30 18:20:27.191645 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.191623 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn" Oct 30 18:20:27.206133 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.206106 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w0-ae3373\"" Oct 30 18:20:27.208790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.208764 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn"] Oct 30 18:20:27.228010 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.227968 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:27.244893 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.244810 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg"] Oct 30 18:20:27.249762 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.249740 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg" Oct 30 18:20:27.262821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.262755 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg"] Oct 30 18:20:27.265332 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.264769 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w0-ae3373-dockercfg-7drmd\"" Oct 30 18:20:27.394537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.391554 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0db16f-42fd-4d0f-8549-33c9d7511b3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg\" (UID: \"fc0db16f-42fd-4d0f-8549-33c9d7511b3d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg" Oct 30 18:20:27.394537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.391699 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9htlc\" (UniqueName: \"kubernetes.io/projected/809fec18-40ef-42e0-9914-ff3b301068c0-kube-api-access-9htlc\") pod \"payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn\" (UID: \"809fec18-40ef-42e0-9914-ff3b301068c0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn" Oct 30 18:20:27.394537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.391830 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809fec18-40ef-42e0-9914-ff3b301068c0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn\" (UID: \"809fec18-40ef-42e0-9914-ff3b301068c0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn" Oct 30 18:20:27.394537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.391931 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6t8fm\" (UniqueName: \"kubernetes.io/projected/fc0db16f-42fd-4d0f-8549-33c9d7511b3d-kube-api-access-6t8fm\") pod \"payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg\" (UID: \"fc0db16f-42fd-4d0f-8549-33c9d7511b3d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg" Oct 30 18:20:27.607586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.591065 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9htlc\" (UniqueName: \"kubernetes.io/projected/809fec18-40ef-42e0-9914-ff3b301068c0-kube-api-access-9htlc\") pod \"payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn\" (UID: \"809fec18-40ef-42e0-9914-ff3b301068c0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn" Oct 30 18:20:27.607586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.591250 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809fec18-40ef-42e0-9914-ff3b301068c0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn\" (UID: \"809fec18-40ef-42e0-9914-ff3b301068c0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn" Oct 30 18:20:27.607586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.591341 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6t8fm\" (UniqueName: \"kubernetes.io/projected/fc0db16f-42fd-4d0f-8549-33c9d7511b3d-kube-api-access-6t8fm\") pod \"payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg\" (UID: \"fc0db16f-42fd-4d0f-8549-33c9d7511b3d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg" Oct 30 18:20:27.607586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.591451 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0db16f-42fd-4d0f-8549-33c9d7511b3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg\" (UID: \"fc0db16f-42fd-4d0f-8549-33c9d7511b3d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg" Oct 30 18:20:27.607586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.592490 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0db16f-42fd-4d0f-8549-33c9d7511b3d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg\" (UID: \"fc0db16f-42fd-4d0f-8549-33c9d7511b3d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg" Oct 30 18:20:27.691340 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.689608 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809fec18-40ef-42e0-9914-ff3b301068c0-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn\" (UID: \"809fec18-40ef-42e0-9914-ff3b301068c0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn" Oct 30 18:20:27.902576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.892635 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9htlc\" (UniqueName: \"kubernetes.io/projected/809fec18-40ef-42e0-9914-ff3b301068c0-kube-api-access-9htlc\") pod \"payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn\" (UID: \"809fec18-40ef-42e0-9914-ff3b301068c0\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn" Oct 30 18:20:27.902576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.893666 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6t8fm\" (UniqueName: \"kubernetes.io/projected/fc0db16f-42fd-4d0f-8549-33c9d7511b3d-kube-api-access-6t8fm\") pod \"payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg\" (UID: \"fc0db16f-42fd-4d0f-8549-33c9d7511b3d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg" Oct 30 18:20:27.903300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.903070 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg" Oct 30 18:20:27.971557 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:20:27.971523 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10b2ac99_f96c_4585_81cb_c28db96b7eb6.slice/crio-5276ab1dab391f2b6902d1a70c13049ea9b48530a504a3029156ed01d1759d31 WatchSource:0}: Error finding container 5276ab1dab391f2b6902d1a70c13049ea9b48530a504a3029156ed01d1759d31: Status 404 returned error can't find the container with id 5276ab1dab391f2b6902d1a70c13049ea9b48530a504a3029156ed01d1759d31 Oct 30 18:20:27.973786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.973757 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w"] Oct 30 18:20:27.974126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.974095 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:20:27.974613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.974176 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:20:27.974613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:27.974221 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:20:28.108239 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:28.108203 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn" Oct 30 18:20:28.239603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:28.239578 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg"] Oct 30 18:20:28.240338 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:20:28.240302 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc0db16f_42fd_4d0f_8549_33c9d7511b3d.slice/crio-1e7a4a3de7bb80b738cf8f3c458814d32953c0805a0ed37fee1b7385a84d23f0 WatchSource:0}: Error finding container 1e7a4a3de7bb80b738cf8f3c458814d32953c0805a0ed37fee1b7385a84d23f0: Status 404 returned error can't find the container with id 1e7a4a3de7bb80b738cf8f3c458814d32953c0805a0ed37fee1b7385a84d23f0 Oct 30 18:20:28.252486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:28.252451 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" event={"ID":"10b2ac99-f96c-4585-81cb-c28db96b7eb6","Type":"ContainerStarted","Data":"91250cd77bf26f98c8ac4967853884efbf17729bffad7675bae9b90b531c4c88"} Oct 30 18:20:28.252640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:28.252529 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" event={"ID":"10b2ac99-f96c-4585-81cb-c28db96b7eb6","Type":"ContainerStarted","Data":"5276ab1dab391f2b6902d1a70c13049ea9b48530a504a3029156ed01d1759d31"} Oct 30 18:20:28.302888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:28.302817 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" podStartSLOduration=2.302802234 podStartE2EDuration="2.302802234s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:28.29207146 +0000 UTC m=+1127.608491783" watchObservedRunningTime="2026-10-30 18:20:28.302802234 +0000 UTC m=+1127.619222555" Oct 30 18:20:28.419788 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:20:28.419760 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod809fec18_40ef_42e0_9914_ff3b301068c0.slice/crio-130dc2afb0e95a94214a22078aede33d56bfe8979c0e5b91cdeeb30a73ac8f66 WatchSource:0}: Error finding container 130dc2afb0e95a94214a22078aede33d56bfe8979c0e5b91cdeeb30a73ac8f66: Status 404 returned error can't find the container with id 130dc2afb0e95a94214a22078aede33d56bfe8979c0e5b91cdeeb30a73ac8f66 Oct 30 18:20:28.422241 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:28.422214 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn"] Oct 30 18:20:29.233688 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:29.233655 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:29.234649 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:29.234611 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:20:29.234748 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:29.234672 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" podUID="10b2ac99-f96c-4585-81cb-c28db96b7eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 18:20:29.264221 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:29.264187 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn" event={"ID":"809fec18-40ef-42e0-9914-ff3b301068c0","Type":"ContainerStarted","Data":"edfd1a9eb5dcf5ea9a117d07a420ce58f04ac17d9d1fe785c79bba184ae86b26"} Oct 30 18:20:29.264377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:29.264249 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn" event={"ID":"809fec18-40ef-42e0-9914-ff3b301068c0","Type":"ContainerStarted","Data":"130dc2afb0e95a94214a22078aede33d56bfe8979c0e5b91cdeeb30a73ac8f66"} Oct 30 18:20:29.264377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:29.264289 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn" Oct 30 18:20:29.266199 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:29.266176 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg" event={"ID":"fc0db16f-42fd-4d0f-8549-33c9d7511b3d","Type":"ContainerStarted","Data":"37bde7ea3551597aa174283c8d326f012a852fa68c5d123407ac3ca861e722a8"} Oct 30 18:20:29.266296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:29.266216 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg" event={"ID":"fc0db16f-42fd-4d0f-8549-33c9d7511b3d","Type":"ContainerStarted","Data":"1e7a4a3de7bb80b738cf8f3c458814d32953c0805a0ed37fee1b7385a84d23f0"} Oct 30 18:20:29.266393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:29.266373 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg" Oct 30 18:20:29.333532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:29.333129 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn" podStartSLOduration=2.333094699 podStartE2EDuration="2.333094699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:29.331040639 +0000 UTC m=+1128.647460960" watchObservedRunningTime="2026-10-30 18:20:29.333094699 +0000 UTC m=+1128.649515021" Oct 30 18:20:29.374774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:29.374721 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg" podStartSLOduration=2.374679906 podStartE2EDuration="2.374679906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:29.371113429 +0000 UTC m=+1128.687533762" watchObservedRunningTime="2026-10-30 18:20:29.374679906 +0000 UTC m=+1128.691100275" Oct 30 18:20:30.230534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:30.229920 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:20:30.230534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:30.229996 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" podUID="10b2ac99-f96c-4585-81cb-c28db96b7eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 18:20:31.232928 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:31.232891 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:20:31.233481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:31.233451 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" podUID="10b2ac99-f96c-4585-81cb-c28db96b7eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 18:20:32.234532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:32.231929 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:20:32.234532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:32.231993 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" podUID="10b2ac99-f96c-4585-81cb-c28db96b7eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 18:20:33.039278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:33.039226 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g"] Oct 30 18:20:33.047501 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:33.047473 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a1c4c4d6-f625g"] Oct 30 18:20:33.230786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:33.230731 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:20:33.231084 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:33.231054 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" podUID="10b2ac99-f96c-4585-81cb-c28db96b7eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 18:20:34.228842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.228809 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" start-of-body= Oct 30 18:20:34.229223 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.228863 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" podUID="10b2ac99-f96c-4585-81cb-c28db96b7eb6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.148:15021/healthz/ready\": dial tcp 10.133.0.148:15021: connect: connection refused" Oct 30 18:20:34.616169 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.616133 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft"] Oct 30 18:20:34.621361 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.621335 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.654794 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.654760 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-fb0abbf7-openshift-default-dockercfg-7qblf\"" Oct 30 18:20:34.655146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.655116 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft"] Oct 30 18:20:34.715026 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.714986 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqtzg\" (UniqueName: \"kubernetes.io/projected/368378dd-a0ab-49f6-8c9d-8f457863b3ec-kube-api-access-qqtzg\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.715187 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.715053 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-envoy\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.715235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.715215 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-credential-socket\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.715268 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.715259 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-workload-socket\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.715321 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.715278 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-podinfo\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.715430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.715408 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-workload-certs\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.715484 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.715438 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-token\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.715562 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.715547 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istiod-ca-cert\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.715619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.715606 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-data\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.836533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.835817 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-workload-certs\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.836533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.835867 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-token\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.836533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.835970 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istiod-ca-cert\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.836533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.836014 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-data\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.836533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.836164 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqtzg\" (UniqueName: \"kubernetes.io/projected/368378dd-a0ab-49f6-8c9d-8f457863b3ec-kube-api-access-qqtzg\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.836533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.836250 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-envoy\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.836533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.836395 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-credential-socket\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.836533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.836449 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-workload-socket\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.836533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.836474 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-podinfo\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.841992 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.841964 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-data\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.857613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.842949 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-workload-certs\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.857849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.843730 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istiod-ca-cert\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.857965 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.843917 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-credential-socket\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.858360 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.858333 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-workload-socket\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.874616 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.867978 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-podinfo\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.912531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.905953 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-envoy\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.912531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.907433 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqtzg\" (UniqueName: \"kubernetes.io/projected/368378dd-a0ab-49f6-8c9d-8f457863b3ec-kube-api-access-qqtzg\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.936538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.934838 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-token\") pod \"e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.960534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.959954 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:34.981987 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:34.974692 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c339c6e-e93b-42c1-860d-b83a3444a58b" path="/var/lib/kubelet/pods/1c339c6e-e93b-42c1-860d-b83a3444a58b/volumes" Oct 30 18:20:35.238795 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:35.237183 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:35.253595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:35.241009 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:35.333792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:35.333722 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:20:35.664382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:35.664352 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft"] Oct 30 18:20:35.665060 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:20:35.665035 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod368378dd_a0ab_49f6_8c9d_8f457863b3ec.slice/crio-1d6ddc7a972e4067cd50c8e97ead030d9cb2307f7c746211086abaad231ba55e WatchSource:0}: Error finding container 1d6ddc7a972e4067cd50c8e97ead030d9cb2307f7c746211086abaad231ba55e: Status 404 returned error can't find the container with id 1d6ddc7a972e4067cd50c8e97ead030d9cb2307f7c746211086abaad231ba55e Oct 30 18:20:35.668402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:35.668367 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:20:35.668501 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:35.668436 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:20:35.668501 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:35.668463 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:20:36.438671 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:36.438629 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" event={"ID":"368378dd-a0ab-49f6-8c9d-8f457863b3ec","Type":"ContainerStarted","Data":"2dbf7f69306cc194d28bf0b85ee1087a7edc9218d2e6de115a878477abb65bb7"} Oct 30 18:20:36.439043 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:36.438695 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" event={"ID":"368378dd-a0ab-49f6-8c9d-8f457863b3ec","Type":"ContainerStarted","Data":"1d6ddc7a972e4067cd50c8e97ead030d9cb2307f7c746211086abaad231ba55e"} Oct 30 18:20:36.507344 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:36.507283 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" podStartSLOduration=2.507264765 podStartE2EDuration="2.507264765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:36.491294808 +0000 UTC m=+1135.807715142" watchObservedRunningTime="2026-10-30 18:20:36.507264765 +0000 UTC m=+1135.823685086" Oct 30 18:20:36.960470 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:36.960437 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:36.965895 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:36.965873 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:37.447471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:37.447440 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:37.448551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:37.448496 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:20:40.278126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:40.278093 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg" Oct 30 18:20:40.278549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:40.278182 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn" Oct 30 18:20:46.273294 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:46.273259 5232 scope.go:122] "RemoveContainer" containerID="d8372a819efa7394c17aa7a9113907387e20c31a66093a98f30bebe2ea22743f" Oct 30 18:20:46.284345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:46.284322 5232 scope.go:122] "RemoveContainer" containerID="21c10db08f62e7afbe59732832398cfe40579dbd24d6ac270195e5c0ca7ac0c7" Oct 30 18:20:46.294728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:46.294709 5232 scope.go:122] "RemoveContainer" containerID="baeebd6b2fc27e7f514f199cdc28d397972e304bb3ad6db19eca64dac327f085" Oct 30 18:20:46.304731 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:46.304713 5232 scope.go:122] "RemoveContainer" containerID="860e0f4b112b9b2f06d5c7726f9fa8ea2f24c6937fd92c6cb736db557e4e47c2" Oct 30 18:20:46.314197 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:46.314180 5232 scope.go:122] "RemoveContainer" containerID="f16ffafe13049216251fa2556cb8160315f4bc95c6ceb91f7408047f91806ccc" Oct 30 18:20:46.324120 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:46.324090 5232 scope.go:122] "RemoveContainer" containerID="7b9c98fd56714eee92b29b449377d34a0959a1cf6b2d8e731dd4397068863c92" Oct 30 18:20:46.333823 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:46.333806 5232 scope.go:122] "RemoveContainer" containerID="d3296c3e674c97821abb98a2da29084860da2cee9349c8870f1d91175661d29a" Oct 30 18:20:46.346892 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:46.346873 5232 scope.go:122] "RemoveContainer" containerID="d2e8532a44453206f628fd71268e51d1df128e7053c1a512368c0c57391925a2" Oct 30 18:20:57.007618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.007584 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh"] Oct 30 18:20:57.012636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.012610 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.037813 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.037784 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh"] Oct 30 18:20:57.043046 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:20:57.043011 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:20:57.134655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.134615 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-token\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.134822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.134691 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-podinfo\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.134822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.134730 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fx7gl\" (UniqueName: \"kubernetes.io/projected/a767e3a9-c897-44b6-87c8-b1be86f89ca7-kube-api-access-fx7gl\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.134908 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.134886 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-credential-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.134948 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.134929 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-data\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.135026 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.135013 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-workload-certs\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.135067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.135039 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istiod-ca-cert\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.135160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.135148 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-workload-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.135219 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.135197 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-envoy\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.236493 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.236449 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istiod-ca-cert\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.236708 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.236627 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-workload-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.236708 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.236683 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-envoy\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.236845 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.236826 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-token\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.236903 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.236894 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-podinfo\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.236957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.236930 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fx7gl\" (UniqueName: \"kubernetes.io/projected/a767e3a9-c897-44b6-87c8-b1be86f89ca7-kube-api-access-fx7gl\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.237018 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.236953 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-workload-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.237157 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.237134 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-credential-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.237283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.237190 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-data\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.237283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.237248 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-workload-certs\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.237283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.237249 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istiod-ca-cert\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.237459 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.237415 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-credential-socket\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.237674 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.237656 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-workload-certs\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.237853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.237830 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-data\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.239180 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.239157 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-envoy\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.239666 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.239644 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-podinfo\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.256389 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.256357 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-token\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.256573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.256552 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fx7gl\" (UniqueName: \"kubernetes.io/projected/a767e3a9-c897-44b6-87c8-b1be86f89ca7-kube-api-access-fx7gl\") pod \"e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.328788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.328712 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:57.657953 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.657867 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh"] Oct 30 18:20:57.662600 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:20:57.662570 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda767e3a9_c897_44b6_87c8_b1be86f89ca7.slice/crio-5295c49e30aec65c2ada0cd00f2a947c76d789ee2dbafee417a0f3d9fb30b401 WatchSource:0}: Error finding container 5295c49e30aec65c2ada0cd00f2a947c76d789ee2dbafee417a0f3d9fb30b401: Status 404 returned error can't find the container with id 5295c49e30aec65c2ada0cd00f2a947c76d789ee2dbafee417a0f3d9fb30b401 Oct 30 18:20:57.665792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.665754 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:20:57.665900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.665835 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:20:57.665900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.665878 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:20:57.999098 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.999062 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" event={"ID":"a767e3a9-c897-44b6-87c8-b1be86f89ca7","Type":"ContainerStarted","Data":"baad1f2a1aa15aa2aaa41c9ea568d972515fb1f520382e3ef537e3a59c7f81f2"} Oct 30 18:20:57.999310 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:57.999130 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" event={"ID":"a767e3a9-c897-44b6-87c8-b1be86f89ca7","Type":"ContainerStarted","Data":"5295c49e30aec65c2ada0cd00f2a947c76d789ee2dbafee417a0f3d9fb30b401"} Oct 30 18:20:58.048591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:58.048503 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" podStartSLOduration=2.04848738 podStartE2EDuration="2.04848738s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:20:58.043185891 +0000 UTC m=+1157.359606225" watchObservedRunningTime="2026-10-30 18:20:58.04848738 +0000 UTC m=+1157.364907701" Oct 30 18:20:58.329233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:58.329138 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:20:58.330661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:58.330629 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 18:20:58.330661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:58.330688 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" podUID="a767e3a9-c897-44b6-87c8-b1be86f89ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 18:20:59.335532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:59.330931 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 18:20:59.335532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:20:59.331006 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" podUID="a767e3a9-c897-44b6-87c8-b1be86f89ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 18:21:00.334934 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:00.334901 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 18:21:00.335140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:00.334968 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" podUID="a767e3a9-c897-44b6-87c8-b1be86f89ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 18:21:01.330534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:01.329936 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 18:21:01.330534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:01.329999 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" podUID="a767e3a9-c897-44b6-87c8-b1be86f89ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 18:21:02.335083 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:02.335035 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 18:21:02.335708 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:02.335114 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" podUID="a767e3a9-c897-44b6-87c8-b1be86f89ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 18:21:03.334694 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:03.334611 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 18:21:03.334694 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:03.334672 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" podUID="a767e3a9-c897-44b6-87c8-b1be86f89ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 18:21:04.329270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:04.329237 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" start-of-body= Oct 30 18:21:04.329732 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:04.329295 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" podUID="a767e3a9-c897-44b6-87c8-b1be86f89ca7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.152:15021/healthz/ready\": dial tcp 10.133.0.152:15021: connect: connection refused" Oct 30 18:21:05.334124 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:05.334097 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:21:05.334647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:05.334378 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:21:05.335242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:05.335226 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:21:06.041693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:06.041653 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r"] Oct 30 18:21:06.125262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:06.125224 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-25a099a4-v5p7r"] Oct 30 18:21:06.806055 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:06.806012 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ea92922-8471-45cf-bcf8-f70b41ead047" path="/var/lib/kubelet/pods/8ea92922-8471-45cf-bcf8-f70b41ead047/volumes" Oct 30 18:21:14.033136 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:14.033099 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw"] Oct 30 18:21:14.045900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:14.045863 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-11e3b4f3-vcnfw"] Oct 30 18:21:14.872602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:14.872559 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07959221-b081-4dec-ab32-2b3adbeee19b" path="/var/lib/kubelet/pods/07959221-b081-4dec-ab32-2b3adbeee19b/volumes" Oct 30 18:21:18.028904 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:18.028867 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92"] Oct 30 18:21:18.041121 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:18.041087 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-58634cfb-d8s92"] Oct 30 18:21:18.804373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:18.804335 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="734d033e-b406-4d56-a09f-ac34c9bf0951" path="/var/lib/kubelet/pods/734d033e-b406-4d56-a09f-ac34c9bf0951/volumes" Oct 30 18:21:20.032214 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:20.032181 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr"] Oct 30 18:21:20.044179 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:20.044149 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d7cf2f23-86lnr"] Oct 30 18:21:20.805012 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:20.804936 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e8606381-4cb4-4096-9365-6e30d42a54ba" path="/var/lib/kubelet/pods/e8606381-4cb4-4096-9365-6e30d42a54ba/volumes" Oct 30 18:21:21.026567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:21.026529 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb"] Oct 30 18:21:21.039679 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:21.039646 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-69dac4e0-rhmjb"] Oct 30 18:21:22.805089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:22.805051 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9bd1fe1d-3eaf-4d4e-857e-caeb89e49083" path="/var/lib/kubelet/pods/9bd1fe1d-3eaf-4d4e-857e-caeb89e49083/volumes" Oct 30 18:21:27.036204 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:21:27.036169 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:21:28.205801 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:28.205757 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b"] Oct 30 18:21:28.303534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:28.279874 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b" Oct 30 18:21:28.308448 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:28.308412 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b"] Oct 30 18:21:28.320370 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:28.319451 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-e5d168\"" Oct 30 18:21:28.320634 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:28.320183 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-e5d168-dockercfg-hlfd9\"" Oct 30 18:21:28.369009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:28.368977 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fwsr\" (UniqueName: \"kubernetes.io/projected/9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2-kube-api-access-6fwsr\") pod \"payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b\" (UID: \"9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b" Oct 30 18:21:28.369204 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:28.369181 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b\" (UID: \"9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b" Oct 30 18:21:28.471475 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:28.471381 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fwsr\" (UniqueName: \"kubernetes.io/projected/9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2-kube-api-access-6fwsr\") pod \"payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b\" (UID: \"9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b" Oct 30 18:21:28.471686 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:28.471639 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b\" (UID: \"9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b" Oct 30 18:21:28.472484 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:28.472460 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b\" (UID: \"9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b" Oct 30 18:21:28.490497 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:28.490464 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fwsr\" (UniqueName: \"kubernetes.io/projected/9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2-kube-api-access-6fwsr\") pod \"payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b\" (UID: \"9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b" Oct 30 18:21:28.633173 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:28.631643 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b" Oct 30 18:21:29.067797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:29.067767 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b"] Oct 30 18:21:29.068840 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:21:29.068814 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c2d4e2f_17e3_4fa7_a9cc_11c74c99d4d2.slice/crio-8f5386ce5962df91ac3d128c077017bf104eb44248c43058cdb48b741905b832 WatchSource:0}: Error finding container 8f5386ce5962df91ac3d128c077017bf104eb44248c43058cdb48b741905b832: Status 404 returned error can't find the container with id 8f5386ce5962df91ac3d128c077017bf104eb44248c43058cdb48b741905b832 Oct 30 18:21:29.494583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:29.494545 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b" event={"ID":"9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2","Type":"ContainerStarted","Data":"43c24ca11c1087509620908cd23a9805ab14bdd5e6db8139dcae25cef92df458"} Oct 30 18:21:29.495005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:29.494602 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b" event={"ID":"9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2","Type":"ContainerStarted","Data":"8f5386ce5962df91ac3d128c077017bf104eb44248c43058cdb48b741905b832"} Oct 30 18:21:29.495005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:29.494822 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b" Oct 30 18:21:29.536077 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:29.536026 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b" podStartSLOduration=2.53601376 podStartE2EDuration="2.53601376s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:21:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:21:29.53270988 +0000 UTC m=+1188.849130205" watchObservedRunningTime="2026-10-30 18:21:29.53601376 +0000 UTC m=+1188.852434080" Oct 30 18:21:40.507029 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:40.506999 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b" Oct 30 18:21:42.808968 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:42.808937 5232 generic.go:356] "Generic (PLEG): container finished" podID="368378dd-a0ab-49f6-8c9d-8f457863b3ec" containerID="2dbf7f69306cc194d28bf0b85ee1087a7edc9218d2e6de115a878477abb65bb7" exitCode=0 Oct 30 18:21:42.809597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:42.809010 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" event={"ID":"368378dd-a0ab-49f6-8c9d-8f457863b3ec","Type":"ContainerDied","Data":"2dbf7f69306cc194d28bf0b85ee1087a7edc9218d2e6de115a878477abb65bb7"} Oct 30 18:21:42.809683 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:42.809633 5232 scope.go:122] "RemoveContainer" containerID="2dbf7f69306cc194d28bf0b85ee1087a7edc9218d2e6de115a878477abb65bb7" Oct 30 18:21:42.811417 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:42.811384 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:21:42.811552 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:42.811458 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:21:42.811552 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:42.811494 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:21:43.843027 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:43.842988 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" event={"ID":"368378dd-a0ab-49f6-8c9d-8f457863b3ec","Type":"ContainerStarted","Data":"48b4a7fe8f0e88fd9fd2b05045a4e31c6b46164bff3895646f5261e88a6afb62"} Oct 30 18:21:43.961479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:43.961448 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:21:43.961666 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:43.961639 5232 patch_prober.go:29] interesting pod/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 30 18:21:43.961717 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:43.961696 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" podUID="368378dd-a0ab-49f6-8c9d-8f457863b3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 30 18:21:44.961443 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:44.961406 5232 patch_prober.go:29] interesting pod/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 30 18:21:44.961973 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:44.961470 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" podUID="368378dd-a0ab-49f6-8c9d-8f457863b3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 30 18:21:45.961309 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:45.961274 5232 patch_prober.go:29] interesting pod/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 30 18:21:45.961600 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:45.961322 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" podUID="368378dd-a0ab-49f6-8c9d-8f457863b3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 30 18:21:46.509074 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:46.509018 5232 scope.go:122] "RemoveContainer" containerID="0203b6e989f1d905c1d841e560c13339287753be4d207f36cdc050c928ff2dbf" Oct 30 18:21:46.960905 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:46.960819 5232 patch_prober.go:29] interesting pod/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 30 18:21:46.960905 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:46.960868 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" podUID="368378dd-a0ab-49f6-8c9d-8f457863b3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 30 18:21:47.067541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:47.060565 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z"] Oct 30 18:21:47.128827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:47.128752 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm"] Oct 30 18:21:47.152348 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:47.152311 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75059e0c-g794z"] Oct 30 18:21:47.159347 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:47.159318 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ff96556-c6wlm"] Oct 30 18:21:47.964813 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:47.964772 5232 patch_prober.go:29] interesting pod/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 30 18:21:47.965115 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:47.965082 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" podUID="368378dd-a0ab-49f6-8c9d-8f457863b3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 30 18:21:48.805598 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:48.805564 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7ff9c66e-0e88-4ce4-9907-09cbd81264ea" path="/var/lib/kubelet/pods/7ff9c66e-0e88-4ce4-9907-09cbd81264ea/volumes" Oct 30 18:21:48.806115 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:48.806075 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c30bde54-a1ab-4ab9-86e1-b3248a4e73b8" path="/var/lib/kubelet/pods/c30bde54-a1ab-4ab9-86e1-b3248a4e73b8/volumes" Oct 30 18:21:48.960654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:48.960623 5232 patch_prober.go:29] interesting pod/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" start-of-body= Oct 30 18:21:48.960839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:48.960674 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" podUID="368378dd-a0ab-49f6-8c9d-8f457863b3ec" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.151:15021/healthz/ready\": dial tcp 10.133.0.151:15021: connect: connection refused" Oct 30 18:21:49.965864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:49.965833 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:21:49.966325 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:49.966300 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:21:49.967030 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:49.967009 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:21:50.035152 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:50.035119 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2"] Oct 30 18:21:50.039999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:50.039973 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-db839263-8g9c2"] Oct 30 18:21:50.805056 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:50.805025 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4f754f7-41c2-48f0-9236-b27c52d968f2" path="/var/lib/kubelet/pods/f4f754f7-41c2-48f0-9236-b27c52d968f2/volumes" Oct 30 18:21:56.027466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:56.027430 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb"] Oct 30 18:21:56.039411 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:56.039380 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b85a6c82-n64pb"] Oct 30 18:21:56.807018 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:56.806979 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c18f5382-cae5-42b9-9b10-b7e6555fd628" path="/var/lib/kubelet/pods/c18f5382-cae5-42b9-9b10-b7e6555fd628/volumes" Oct 30 18:21:57.036048 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:21:57.036011 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:21:58.032423 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:58.032386 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd"] Oct 30 18:21:58.043844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:58.043815 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5d67033e-b98kd"] Oct 30 18:21:58.805540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:21:58.805489 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72f09463-8af4-4ed2-b185-189878a28c11" path="/var/lib/kubelet/pods/72f09463-8af4-4ed2-b185-189878a28c11/volumes" Oct 30 18:22:08.030346 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:08.030306 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx"] Oct 30 18:22:08.041705 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:08.041679 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fd0cd224-8z6cx"] Oct 30 18:22:08.804479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:08.804447 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="672af468-d0e0-4276-b532-add124f8b2a5" path="/var/lib/kubelet/pods/672af468-d0e0-4276-b532-add124f8b2a5/volumes" Oct 30 18:22:09.029676 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:09.029640 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g"] Oct 30 18:22:09.041561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:09.041532 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28f0a739-jbb8g"] Oct 30 18:22:10.030216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:10.030185 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9"] Oct 30 18:22:10.042212 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:10.042183 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c105a403-6cks9"] Oct 30 18:22:10.805612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:10.805584 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="39d2382d-51b2-4f31-aa8e-05205da06225" path="/var/lib/kubelet/pods/39d2382d-51b2-4f31-aa8e-05205da06225/volumes" Oct 30 18:22:10.806085 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:10.806070 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95" path="/var/lib/kubelet/pods/beb8bf6d-8c9f-43d0-8a35-ea3ab5c24c95/volumes" Oct 30 18:22:12.099663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:12.099603 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7"] Oct 30 18:22:12.100254 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:12.100231 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a904b6fd-59kj7"] Oct 30 18:22:12.807835 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:12.807799 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0b82ac4-2aae-47c4-b032-ab5e463a7920" path="/var/lib/kubelet/pods/e0b82ac4-2aae-47c4-b032-ab5e463a7920/volumes" Oct 30 18:22:14.026238 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:14.026202 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn"] Oct 30 18:22:14.037494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:14.037468 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0c6a9955-vwczn"] Oct 30 18:22:14.805100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:14.805070 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81cb79ad-89d4-498e-be40-10ee94aa6861" path="/var/lib/kubelet/pods/81cb79ad-89d4-498e-be40-10ee94aa6861/volumes" Oct 30 18:22:20.027825 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.027788 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7"] Oct 30 18:22:20.039247 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.039208 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-fdef74e1-9dpb7"] Oct 30 18:22:20.574160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.574119 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z"] Oct 30 18:22:20.579242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.579214 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.597772 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.597744 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-ae3373-openshift-default-dockercfg-kcj2c\"" Oct 30 18:22:20.620744 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.620711 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z"] Oct 30 18:22:20.680796 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.680765 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rrr6d\" (UniqueName: \"kubernetes.io/projected/24aaf415-4614-4e59-87db-71cac71da3d9-kube-api-access-rrr6d\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.680974 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.680903 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24aaf415-4614-4e59-87db-71cac71da3d9-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.680974 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.680957 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.681092 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.681076 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.681146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.681133 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24aaf415-4614-4e59-87db-71cac71da3d9-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.681185 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.681166 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24aaf415-4614-4e59-87db-71cac71da3d9-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.681222 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.681188 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.681222 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.681213 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.681285 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.681238 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.782983 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.782951 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.783161 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.783015 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24aaf415-4614-4e59-87db-71cac71da3d9-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.783161 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.783047 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24aaf415-4614-4e59-87db-71cac71da3d9-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.783161 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.783071 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.783161 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.783094 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.783161 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.783123 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.783435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.783310 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rrr6d\" (UniqueName: \"kubernetes.io/projected/24aaf415-4614-4e59-87db-71cac71da3d9-kube-api-access-rrr6d\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.783599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.783567 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.783739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.783637 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.783739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.783669 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.783739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.783677 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24aaf415-4614-4e59-87db-71cac71da3d9-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.784090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.783747 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.784090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.783933 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24aaf415-4614-4e59-87db-71cac71da3d9-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.784090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.783999 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.785772 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.785754 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24aaf415-4614-4e59-87db-71cac71da3d9-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.785911 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.785890 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.802560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.802490 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rrr6d\" (UniqueName: \"kubernetes.io/projected/24aaf415-4614-4e59-87db-71cac71da3d9-kube-api-access-rrr6d\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.802701 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.802594 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24aaf415-4614-4e59-87db-71cac71da3d9-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:20.805336 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.805311 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82dbd158-2d7e-4c05-9933-43d6e8b85a7d" path="/var/lib/kubelet/pods/82dbd158-2d7e-4c05-9933-43d6e8b85a7d/volumes" Oct 30 18:22:20.895721 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:20.895646 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:21.214461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.214432 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z"] Oct 30 18:22:21.216655 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:22:21.216622 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24aaf415_4614_4e59_87db_71cac71da3d9.slice/crio-a2fc79d7364d5fae04cdc807760f63722c36728f157220b9569db75e6dd4063e WatchSource:0}: Error finding container a2fc79d7364d5fae04cdc807760f63722c36728f157220b9569db75e6dd4063e: Status 404 returned error can't find the container with id a2fc79d7364d5fae04cdc807760f63722c36728f157220b9569db75e6dd4063e Oct 30 18:22:21.219652 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.219618 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:22:21.219731 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.219694 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:22:21.219731 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.219726 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:22:21.518580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.516285 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" event={"ID":"24aaf415-4614-4e59-87db-71cac71da3d9","Type":"ContainerStarted","Data":"9c51eb8456d05e5e6c2a23831daf3279f69d249bad8c82a3aac8f93a0171a1b6"} Oct 30 18:22:21.518580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.516365 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" event={"ID":"24aaf415-4614-4e59-87db-71cac71da3d9","Type":"ContainerStarted","Data":"a2fc79d7364d5fae04cdc807760f63722c36728f157220b9569db75e6dd4063e"} Oct 30 18:22:21.562459 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.562399 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" podStartSLOduration=1.5623821 podStartE2EDuration="1.5623821s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:21.558032736 +0000 UTC m=+1240.874453057" watchObservedRunningTime="2026-10-30 18:22:21.5623821 +0000 UTC m=+1240.878802488" Oct 30 18:22:21.615432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.615394 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x"] Oct 30 18:22:21.620396 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.620370 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.637437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.637405 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f8709f-openshift-default-dockercfg-5ptgl\"" Oct 30 18:22:21.641725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.641561 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x"] Oct 30 18:22:21.698447 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.698412 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961c8dee-4505-4b84-b984-3006b6a95dd8-istiod-ca-cert\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.698688 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.698663 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-envoy\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.698809 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.698786 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-data\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.698861 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.698842 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzb5v\" (UniqueName: \"kubernetes.io/projected/961c8dee-4505-4b84-b984-3006b6a95dd8-kube-api-access-dzb5v\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.699028 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.699008 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-workload-certs\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.699186 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.699063 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-credential-socket\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.699186 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.699090 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-podinfo\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.699186 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.699145 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-token\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.699432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.699410 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-workload-socket\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.820640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.805968 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-workload-certs\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.820640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.806087 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-credential-socket\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.820640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.806120 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-podinfo\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.820640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.806202 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-token\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.820640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.806352 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-workload-socket\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.820640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.806452 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961c8dee-4505-4b84-b984-3006b6a95dd8-istiod-ca-cert\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.820640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.806604 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-envoy\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.820640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.806676 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-data\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.820640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.814136 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-workload-certs\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.820640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.814419 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-credential-socket\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.867717 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.828677 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961c8dee-4505-4b84-b984-3006b6a95dd8-istiod-ca-cert\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.867717 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.838134 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-workload-socket\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.867717 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.848719 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-data\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.867717 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.848839 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzb5v\" (UniqueName: \"kubernetes.io/projected/961c8dee-4505-4b84-b984-3006b6a95dd8-kube-api-access-dzb5v\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.910537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.900853 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:21.927143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.922650 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-podinfo\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.927143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.924830 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-token\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:21.934122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.934013 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-envoy\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:22.010537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.965578 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc"] Oct 30 18:22:22.010537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:21.993056 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc" Oct 30 18:22:22.043592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.020260 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:22:22.043592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.020353 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:22:22.116593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.106703 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc"] Oct 30 18:22:22.288440 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.225006 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzb5v\" (UniqueName: \"kubernetes.io/projected/961c8dee-4505-4b84-b984-3006b6a95dd8-kube-api-access-dzb5v\") pod \"e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:22.288440 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.244809 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:22.386018 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.385906 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4j7b6\" (UniqueName: \"kubernetes.io/projected/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-kube-api-access-4j7b6\") pod \"maas-api-revoke-sub-7cbf63e5-8jnvc\" (UID: \"79cb5f7a-0cbb-420f-886b-cd55e96f16ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc" Oct 30 18:22:22.386249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.386111 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cbf63e5-8jnvc\" (UID: \"79cb5f7a-0cbb-420f-886b-cd55e96f16ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc" Oct 30 18:22:22.386315 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.386244 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cbf63e5-8jnvc\" (UID: \"79cb5f7a-0cbb-420f-886b-cd55e96f16ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc" Oct 30 18:22:22.488717 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.488680 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4j7b6\" (UniqueName: \"kubernetes.io/projected/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-kube-api-access-4j7b6\") pod \"maas-api-revoke-sub-7cbf63e5-8jnvc\" (UID: \"79cb5f7a-0cbb-420f-886b-cd55e96f16ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc" Oct 30 18:22:22.488876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.488834 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cbf63e5-8jnvc\" (UID: \"79cb5f7a-0cbb-420f-886b-cd55e96f16ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc" Oct 30 18:22:22.488932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.488917 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cbf63e5-8jnvc\" (UID: \"79cb5f7a-0cbb-420f-886b-cd55e96f16ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc" Oct 30 18:22:22.490179 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.490158 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7cbf63e5-8jnvc\" (UID: \"79cb5f7a-0cbb-420f-886b-cd55e96f16ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc" Oct 30 18:22:22.492272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.492251 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7cbf63e5-8jnvc\" (UID: \"79cb5f7a-0cbb-420f-886b-cd55e96f16ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc" Oct 30 18:22:22.508693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.508664 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4j7b6\" (UniqueName: \"kubernetes.io/projected/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-kube-api-access-4j7b6\") pod \"maas-api-revoke-sub-7cbf63e5-8jnvc\" (UID: \"79cb5f7a-0cbb-420f-886b-cd55e96f16ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc" Oct 30 18:22:22.696991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.696912 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc" Oct 30 18:22:22.752581 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:22:22.752547 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod961c8dee_4505_4b84_b984_3006b6a95dd8.slice/crio-bb83f5dce8a9b6e0f30708dc978a46039ae0f61c5fdb5bfb91db606a68e54453 WatchSource:0}: Error finding container bb83f5dce8a9b6e0f30708dc978a46039ae0f61c5fdb5bfb91db606a68e54453: Status 404 returned error can't find the container with id bb83f5dce8a9b6e0f30708dc978a46039ae0f61c5fdb5bfb91db606a68e54453 Oct 30 18:22:22.754901 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.754866 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x"] Oct 30 18:22:22.755960 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.755925 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:22:22.756043 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.755998 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:22:22.756043 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.756029 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:22:22.897141 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.897111 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 18:22:22.897247 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:22.897167 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" podUID="24aaf415-4614-4e59-87db-71cac71da3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 18:22:23.041790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:23.041759 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc"] Oct 30 18:22:23.042995 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:22:23.042966 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79cb5f7a_0cbb_420f_886b_cd55e96f16ee.slice/crio-211f818c84428d30f9af0762bf2da54db8ee600085a9d3f1963ec7e03b350274 WatchSource:0}: Error finding container 211f818c84428d30f9af0762bf2da54db8ee600085a9d3f1963ec7e03b350274: Status 404 returned error can't find the container with id 211f818c84428d30f9af0762bf2da54db8ee600085a9d3f1963ec7e03b350274 Oct 30 18:22:23.674598 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:23.666099 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc" event={"ID":"79cb5f7a-0cbb-420f-886b-cd55e96f16ee","Type":"ContainerStarted","Data":"211f818c84428d30f9af0762bf2da54db8ee600085a9d3f1963ec7e03b350274"} Oct 30 18:22:23.674598 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:23.669844 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" event={"ID":"961c8dee-4505-4b84-b984-3006b6a95dd8","Type":"ContainerStarted","Data":"615e3e5a46e56b54b6f6be9d6d6d7cccaa5dc27727a25838ad2b231bf448ab3e"} Oct 30 18:22:23.674598 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:23.669893 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" event={"ID":"961c8dee-4505-4b84-b984-3006b6a95dd8","Type":"ContainerStarted","Data":"bb83f5dce8a9b6e0f30708dc978a46039ae0f61c5fdb5bfb91db606a68e54453"} Oct 30 18:22:23.766879 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:23.766825 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" podStartSLOduration=2.766810284 podStartE2EDuration="2.766810284s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:23.763811379 +0000 UTC m=+1243.080231701" watchObservedRunningTime="2026-10-30 18:22:23.766810284 +0000 UTC m=+1243.083230605" Oct 30 18:22:23.936598 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:23.930685 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 18:22:23.936598 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:23.930752 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" podUID="24aaf415-4614-4e59-87db-71cac71da3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 18:22:24.245909 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:24.245878 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:24.252269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:24.252242 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:24.679437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:24.679344 5232 generic.go:356] "Generic (PLEG): container finished" podID="79cb5f7a-0cbb-420f-886b-cd55e96f16ee" containerID="c91ddb1d1cc92bcc4a5950e1faffadaa0732735c9d435738715aaef6c4f229ec" exitCode=0 Oct 30 18:22:24.679437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:24.679410 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc" event={"ID":"79cb5f7a-0cbb-420f-886b-cd55e96f16ee","Type":"ContainerDied","Data":"c91ddb1d1cc92bcc4a5950e1faffadaa0732735c9d435738715aaef6c4f229ec"} Oct 30 18:22:24.680026 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:24.679786 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:24.680714 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:24.680696 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:22:24.896593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:24.896563 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 18:22:24.896777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:24.896611 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" podUID="24aaf415-4614-4e59-87db-71cac71da3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 18:22:25.896070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:25.896038 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 18:22:25.896458 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:25.896088 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" podUID="24aaf415-4614-4e59-87db-71cac71da3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 18:22:26.083927 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.083898 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc" Oct 30 18:22:26.170084 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.169949 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-kube-api-access-4j7b6\" (UniqueName: \"kubernetes.io/projected/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-kube-api-access-4j7b6\") pod \"79cb5f7a-0cbb-420f-886b-cd55e96f16ee\" (UID: \"79cb5f7a-0cbb-420f-886b-cd55e96f16ee\") " Oct 30 18:22:26.170261 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.170097 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-maas-api-openshift-ca\") pod \"79cb5f7a-0cbb-420f-886b-cd55e96f16ee\" (UID: \"79cb5f7a-0cbb-420f-886b-cd55e96f16ee\") " Oct 30 18:22:26.170408 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.170343 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-maas-api-xks-ca\") pod \"79cb5f7a-0cbb-420f-886b-cd55e96f16ee\" (UID: \"79cb5f7a-0cbb-420f-886b-cd55e96f16ee\") " Oct 30 18:22:26.172188 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.172146 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-maas-api-openshift-ca" pod "79cb5f7a-0cbb-420f-886b-cd55e96f16ee" (UID: "79cb5f7a-0cbb-420f-886b-cd55e96f16ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:26.175828 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.174445 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-maas-api-xks-ca" pod "79cb5f7a-0cbb-420f-886b-cd55e96f16ee" (UID: "79cb5f7a-0cbb-420f-886b-cd55e96f16ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:22:26.180006 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.179983 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-kube-api-access-4j7b6" pod "79cb5f7a-0cbb-420f-886b-cd55e96f16ee" (UID: "79cb5f7a-0cbb-420f-886b-cd55e96f16ee"). InnerVolumeSpecName "kube-api-access-4j7b6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:26.273320 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.273286 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.273320 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.273318 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4j7b6\" (UniqueName: \"kubernetes.io/projected/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-kube-api-access-4j7b6\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.273320 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.273330 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/79cb5f7a-0cbb-420f-886b-cd55e96f16ee-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:22:26.772961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.772930 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc" Oct 30 18:22:26.773152 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.772967 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc" event={"ID":"79cb5f7a-0cbb-420f-886b-cd55e96f16ee","Type":"ContainerDied","Data":"211f818c84428d30f9af0762bf2da54db8ee600085a9d3f1963ec7e03b350274"} Oct 30 18:22:26.773152 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.773036 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="211f818c84428d30f9af0762bf2da54db8ee600085a9d3f1963ec7e03b350274" Oct 30 18:22:26.896168 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.896133 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 18:22:26.896589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.896184 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" podUID="24aaf415-4614-4e59-87db-71cac71da3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 18:22:26.954882 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.954846 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9"] Oct 30 18:22:26.955859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.955837 5232 cpu_manager.go:414] "containerMap: removing container" podUID="79cb5f7a-0cbb-420f-886b-cd55e96f16ee" containerName="revoke-keys" Oct 30 18:22:26.955859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.955857 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="79cb5f7a-0cbb-420f-886b-cd55e96f16ee" containerName="revoke-keys" Oct 30 18:22:26.956024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.955990 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="79cb5f7a-0cbb-420f-886b-cd55e96f16ee" containerName="revoke-keys" Oct 30 18:22:26.959115 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.959094 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:26.983971 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.983931 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9"] Oct 30 18:22:26.984894 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.984862 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:26.985040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.984916 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:26.985110 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.985092 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:26.985180 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.985109 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:26.985180 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.985147 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:26.985180 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.985163 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3eac53-5e79-43d3-8965-d45420dd403b-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:26.985336 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.985194 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:26.985336 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.985223 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7xq4n\" (UniqueName: \"kubernetes.io/projected/9c3eac53-5e79-43d3-8965-d45420dd403b-kube-api-access-7xq4n\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:26.985336 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:26.985294 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.043611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.043218 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv"] Oct 30 18:22:27.052190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.052144 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2f190c19-9ndlv"] Oct 30 18:22:27.055896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.055862 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w0-e5d168-openshift-default-dockercfg-hlbp7\"" Oct 30 18:22:27.080698 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:22:27.080665 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:22:27.086843 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.086811 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.086995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.086848 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.086995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.086897 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.086995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.086921 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3eac53-5e79-43d3-8965-d45420dd403b-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.086995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.086968 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.087221 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.087001 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7xq4n\" (UniqueName: \"kubernetes.io/projected/9c3eac53-5e79-43d3-8965-d45420dd403b-kube-api-access-7xq4n\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.087221 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.087111 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.087322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.087244 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.087322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.087283 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.087322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.087295 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.087322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.087298 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.087714 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.087694 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3eac53-5e79-43d3-8965-d45420dd403b-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.087869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.087853 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.087978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.087957 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.089465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.089445 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.090295 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.090271 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.135957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.135262 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7xq4n\" (UniqueName: \"kubernetes.io/projected/9c3eac53-5e79-43d3-8965-d45420dd403b-kube-api-access-7xq4n\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.137924 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.137898 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.274603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.274566 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:27.731608 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.703895 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9"] Oct 30 18:22:27.809361 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.809316 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:22:27.809462 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.809417 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:22:27.814624 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.809459 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:22:27.846391 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.846363 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" event={"ID":"9c3eac53-5e79-43d3-8965-d45420dd403b","Type":"ContainerStarted","Data":"18946b6b3de242245d86f2aebd436bdec2d1ad113836242ccc70d4eeb5ab36b9"} Oct 30 18:22:27.896117 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.896077 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" start-of-body= Oct 30 18:22:27.896265 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:27.896136 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" podUID="24aaf415-4614-4e59-87db-71cac71da3d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.154:15021/healthz/ready\": dial tcp 10.133.0.154:15021: connect: connection refused" Oct 30 18:22:28.113686 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.113645 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq"] Oct 30 18:22:28.118055 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.118026 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq" Oct 30 18:22:28.133323 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.133260 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f8709f\"" Oct 30 18:22:28.133565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.133493 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq"] Oct 30 18:22:28.228374 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.228322 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smhp5\" (UniqueName: \"kubernetes.io/projected/4cf43950-0cdf-489c-9fd6-95ef1bb93cea-kube-api-access-smhp5\") pod \"payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq\" (UID: \"4cf43950-0cdf-489c-9fd6-95ef1bb93cea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq" Oct 30 18:22:28.234548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.230830 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f8709f-dockercfg-ztnkv\"" Oct 30 18:22:28.450773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.450673 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf43950-0cdf-489c-9fd6-95ef1bb93cea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq\" (UID: \"4cf43950-0cdf-489c-9fd6-95ef1bb93cea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq" Oct 30 18:22:28.530535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.510007 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg"] Oct 30 18:22:28.630383 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.623303 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smhp5\" (UniqueName: \"kubernetes.io/projected/4cf43950-0cdf-489c-9fd6-95ef1bb93cea-kube-api-access-smhp5\") pod \"payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq\" (UID: \"4cf43950-0cdf-489c-9fd6-95ef1bb93cea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq" Oct 30 18:22:28.630383 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.623779 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf43950-0cdf-489c-9fd6-95ef1bb93cea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq\" (UID: \"4cf43950-0cdf-489c-9fd6-95ef1bb93cea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq" Oct 30 18:22:28.743269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.740101 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf43950-0cdf-489c-9fd6-95ef1bb93cea-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq\" (UID: \"4cf43950-0cdf-489c-9fd6-95ef1bb93cea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq" Oct 30 18:22:28.744080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.743820 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg" Oct 30 18:22:28.758736 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.758696 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg"] Oct 30 18:22:28.780590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.779988 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smhp5\" (UniqueName: \"kubernetes.io/projected/4cf43950-0cdf-489c-9fd6-95ef1bb93cea-kube-api-access-smhp5\") pod \"payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq\" (UID: \"4cf43950-0cdf-489c-9fd6-95ef1bb93cea\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq" Oct 30 18:22:28.805589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.805560 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dd02103a-73fc-4e8d-b77d-128241044320" path="/var/lib/kubelet/pods/dd02103a-73fc-4e8d-b77d-128241044320/volumes" Oct 30 18:22:28.861579 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.861500 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" event={"ID":"9c3eac53-5e79-43d3-8965-d45420dd403b","Type":"ContainerStarted","Data":"dad2532a7de8909d7b098a148fde8fa42f2b98acc783b10bbee6992981d6a09c"} Oct 30 18:22:28.874656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.874625 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9cncf\" (UniqueName: \"kubernetes.io/projected/60ace843-6055-4728-a15f-6c6f65c449c3-kube-api-access-9cncf\") pod \"payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg\" (UID: \"60ace843-6055-4728-a15f-6c6f65c449c3\") " pod="openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg" Oct 30 18:22:28.874965 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.874940 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ace843-6055-4728-a15f-6c6f65c449c3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg\" (UID: \"60ace843-6055-4728-a15f-6c6f65c449c3\") " pod="openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg" Oct 30 18:22:28.902090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.902064 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:28.902590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.902334 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:28.903245 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.903221 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:22:28.909734 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.909683 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" podStartSLOduration=2.909669963 podStartE2EDuration="2.909669963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:28.90396021 +0000 UTC m=+1248.220380534" watchObservedRunningTime="2026-10-30 18:22:28.909669963 +0000 UTC m=+1248.226090284" Oct 30 18:22:28.977298 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.977243 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9cncf\" (UniqueName: \"kubernetes.io/projected/60ace843-6055-4728-a15f-6c6f65c449c3-kube-api-access-9cncf\") pod \"payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg\" (UID: \"60ace843-6055-4728-a15f-6c6f65c449c3\") " pod="openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg" Oct 30 18:22:28.977752 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.977727 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ace843-6055-4728-a15f-6c6f65c449c3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg\" (UID: \"60ace843-6055-4728-a15f-6c6f65c449c3\") " pod="openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg" Oct 30 18:22:28.981469 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.981396 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ace843-6055-4728-a15f-6c6f65c449c3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg\" (UID: \"60ace843-6055-4728-a15f-6c6f65c449c3\") " pod="openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg" Oct 30 18:22:28.997237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:28.997135 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9cncf\" (UniqueName: \"kubernetes.io/projected/60ace843-6055-4728-a15f-6c6f65c449c3-kube-api-access-9cncf\") pod \"payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg\" (UID: \"60ace843-6055-4728-a15f-6c6f65c449c3\") " pod="openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg" Oct 30 18:22:29.049057 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:29.048390 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq" Oct 30 18:22:29.103008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:29.102179 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg" Oct 30 18:22:29.275245 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:29.275217 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:29.276439 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:29.276286 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 18:22:29.278335 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:29.278013 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" podUID="9c3eac53-5e79-43d3-8965-d45420dd403b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 18:22:29.452821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:29.452785 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq"] Oct 30 18:22:29.492063 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:29.491466 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg"] Oct 30 18:22:29.495837 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:22:29.495809 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60ace843_6055_4728_a15f_6c6f65c449c3.slice/crio-76a2c74be401b703a3f465d5ab43f9b728001c0fd587a7dab74bf52342ab07f8 WatchSource:0}: Error finding container 76a2c74be401b703a3f465d5ab43f9b728001c0fd587a7dab74bf52342ab07f8: Status 404 returned error can't find the container with id 76a2c74be401b703a3f465d5ab43f9b728001c0fd587a7dab74bf52342ab07f8 Oct 30 18:22:29.878325 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:29.878235 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg" event={"ID":"60ace843-6055-4728-a15f-6c6f65c449c3","Type":"ContainerStarted","Data":"18aee18810fc882e5ca7e16d2acc6fb558eded02d68e1ab8f4289b943583aca0"} Oct 30 18:22:29.878889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:29.878872 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg" Oct 30 18:22:29.879386 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:29.879354 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg" event={"ID":"60ace843-6055-4728-a15f-6c6f65c449c3","Type":"ContainerStarted","Data":"76a2c74be401b703a3f465d5ab43f9b728001c0fd587a7dab74bf52342ab07f8"} Oct 30 18:22:29.880863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:29.880806 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq" event={"ID":"4cf43950-0cdf-489c-9fd6-95ef1bb93cea","Type":"ContainerStarted","Data":"413eaa14c676bef4fb87b40157921525c83f853188c490f07bf5c382dbb1f3d9"} Oct 30 18:22:29.880863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:29.880843 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq" event={"ID":"4cf43950-0cdf-489c-9fd6-95ef1bb93cea","Type":"ContainerStarted","Data":"a9ede13ff34aea8d9c67bada1b08f19f63f15a587c0de82599aa6476f9d6ca93"} Oct 30 18:22:29.943561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:29.943119 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg" podStartSLOduration=1.9430709510000002 podStartE2EDuration="1.943070951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:29.929224643 +0000 UTC m=+1249.245644965" watchObservedRunningTime="2026-10-30 18:22:29.943070951 +0000 UTC m=+1249.259491279" Oct 30 18:22:29.989469 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:29.989403 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq" podStartSLOduration=1.9893852330000001 podStartE2EDuration="1.989385233s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:29.986602584 +0000 UTC m=+1249.303022905" watchObservedRunningTime="2026-10-30 18:22:29.989385233 +0000 UTC m=+1249.305805558" Oct 30 18:22:30.275647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:30.275265 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 18:22:30.275647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:30.275322 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" podUID="9c3eac53-5e79-43d3-8965-d45420dd403b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 18:22:30.963544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:30.958713 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq" Oct 30 18:22:31.290612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:31.279700 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 18:22:31.290612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:31.279763 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" podUID="9c3eac53-5e79-43d3-8965-d45420dd403b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 18:22:32.356543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:32.331682 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 18:22:32.356543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:32.331771 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" podUID="9c3eac53-5e79-43d3-8965-d45420dd403b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 18:22:33.275112 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:33.275072 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 18:22:33.275342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:33.275131 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" podUID="9c3eac53-5e79-43d3-8965-d45420dd403b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 18:22:34.275427 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:34.275397 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" start-of-body= Oct 30 18:22:34.275881 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:34.275447 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" podUID="9c3eac53-5e79-43d3-8965-d45420dd403b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.157:15021/healthz/ready\": dial tcp 10.133.0.157:15021: connect: connection refused" Oct 30 18:22:35.280031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.279998 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:35.280543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.280390 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:35.281054 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.281033 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:22:35.641717 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.641630 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24"] Oct 30 18:22:35.645877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.645855 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.669128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.669100 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24"] Oct 30 18:22:35.688119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.688073 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw"] Oct 30 18:22:35.721718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.721687 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.734367 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.734332 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.735593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.734386 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.735593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.734473 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.735593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.734524 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xrk72\" (UniqueName: \"kubernetes.io/projected/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-kube-api-access-xrk72\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.735593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.734558 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.735593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.734598 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.735593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.734681 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.735593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.734714 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.735593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.734885 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.737658 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.737628 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw"] Oct 30 18:22:35.836443 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.836410 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.836699 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.836450 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xrk72\" (UniqueName: \"kubernetes.io/projected/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-kube-api-access-xrk72\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.836699 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.836477 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.836699 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.836499 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.836699 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.836542 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.836699 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.836576 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7trt4\" (UniqueName: \"kubernetes.io/projected/37acc5cd-e689-4f1f-bb72-38794eb5e205-kube-api-access-7trt4\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.836995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.836714 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.836995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.836735 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.836995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.836787 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.836995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.836821 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.836995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.836841 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.836995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.836947 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37acc5cd-e689-4f1f-bb72-38794eb5e205-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.836995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.836985 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.837235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.837001 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.837235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.837067 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.837235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.837110 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.837235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.837187 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.837384 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.837281 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.837384 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.837314 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.837491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.837448 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.837617 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.837582 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.837617 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.837586 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.837956 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.837932 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.839648 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.839622 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.839797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.839774 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.859333 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.859282 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.859843 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.859819 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xrk72\" (UniqueName: \"kubernetes.io/projected/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-kube-api-access-xrk72\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:35.939470 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.939389 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.939470 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.939433 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7trt4\" (UniqueName: \"kubernetes.io/projected/37acc5cd-e689-4f1f-bb72-38794eb5e205-kube-api-access-7trt4\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.939727 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.939495 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.939727 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.939597 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.939727 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.939634 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.939727 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.939716 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37acc5cd-e689-4f1f-bb72-38794eb5e205-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.939961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.939805 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.939961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.939844 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.940073 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.939960 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.940073 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.940046 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.940073 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.940051 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.940239 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.940070 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.940239 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.940156 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.940599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.940572 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37acc5cd-e689-4f1f-bb72-38794eb5e205-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.942797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.942774 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.942900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.942839 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.960545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.960498 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7trt4\" (UniqueName: \"kubernetes.io/projected/37acc5cd-e689-4f1f-bb72-38794eb5e205-kube-api-access-7trt4\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.961160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.961140 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:35.964776 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:35.964750 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:36.040684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:36.040650 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:36.098366 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:36.098331 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9"] Oct 30 18:22:36.115149 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:36.115117 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6"] Oct 30 18:22:36.125043 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:36.124997 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e8ac6bc1-7q9n9"] Oct 30 18:22:36.133030 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:36.132991 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bfd542ec-qmpd6"] Oct 30 18:22:36.312271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:36.312241 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24"] Oct 30 18:22:36.312737 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:22:36.312584 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f5b3876_dc3b_4f39_a6f1_4591ce4a9998.slice/crio-62f9a3fc83a586a639e08fc877f239337a28bdc41956b644fb879e195de3fc1a WatchSource:0}: Error finding container 62f9a3fc83a586a639e08fc877f239337a28bdc41956b644fb879e195de3fc1a: Status 404 returned error can't find the container with id 62f9a3fc83a586a639e08fc877f239337a28bdc41956b644fb879e195de3fc1a Oct 30 18:22:36.315699 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:36.315651 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:22:36.315833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:36.315756 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:22:36.315833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:36.315793 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:22:36.390846 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:36.388587 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw"] Oct 30 18:22:36.392647 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:22:36.392598 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37acc5cd_e689_4f1f_bb72_38794eb5e205.slice/crio-806ddbade2a1c1b63ede905361c73c8f27f54f3c7fa0bf537c89c0fc011a85fb WatchSource:0}: Error finding container 806ddbade2a1c1b63ede905361c73c8f27f54f3c7fa0bf537c89c0fc011a85fb: Status 404 returned error can't find the container with id 806ddbade2a1c1b63ede905361c73c8f27f54f3c7fa0bf537c89c0fc011a85fb Oct 30 18:22:36.396487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:36.396447 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:22:36.396625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:36.396553 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:22:36.396625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:36.396595 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:22:36.812423 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:36.812389 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97736456-4f31-4954-94f2-f9500b85dc8c" path="/var/lib/kubelet/pods/97736456-4f31-4954-94f2-f9500b85dc8c/volumes" Oct 30 18:22:36.815334 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:36.815308 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa62fecb-fdca-4f29-b713-0bfbb78178f9" path="/var/lib/kubelet/pods/aa62fecb-fdca-4f29-b713-0bfbb78178f9/volumes" Oct 30 18:22:37.099592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:37.094156 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" event={"ID":"37acc5cd-e689-4f1f-bb72-38794eb5e205","Type":"ContainerStarted","Data":"0e97b95f73eecec35274b0726fc5500f57a6051ebfa5dc281f87edf566737e47"} Oct 30 18:22:37.099592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:37.094247 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" event={"ID":"37acc5cd-e689-4f1f-bb72-38794eb5e205","Type":"ContainerStarted","Data":"806ddbade2a1c1b63ede905361c73c8f27f54f3c7fa0bf537c89c0fc011a85fb"} Oct 30 18:22:37.117878 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:37.117846 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" event={"ID":"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998","Type":"ContainerStarted","Data":"c20ab9c1c9576eb5e86497bb8ec8e3ccd1787a00cfb6798231e7d1bb123f8455"} Oct 30 18:22:37.118160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:37.118136 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" event={"ID":"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998","Type":"ContainerStarted","Data":"62f9a3fc83a586a639e08fc877f239337a28bdc41956b644fb879e195de3fc1a"} Oct 30 18:22:37.255191 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:37.253413 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" podStartSLOduration=2.253393352 podStartE2EDuration="2.253393352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:37.164179837 +0000 UTC m=+1256.480600160" watchObservedRunningTime="2026-10-30 18:22:37.253393352 +0000 UTC m=+1256.569813674" Oct 30 18:22:37.266549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:37.266449 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" podStartSLOduration=2.266390808 podStartE2EDuration="2.266390808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:22:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:22:37.249454154 +0000 UTC m=+1256.565874471" watchObservedRunningTime="2026-10-30 18:22:37.266390808 +0000 UTC m=+1256.582811128" Oct 30 18:22:37.968372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:37.965938 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:38.047539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:38.043007 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:38.093536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:38.092607 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:38.169015 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:38.166842 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:38.300108 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:38.296089 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:38.300108 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:38.296162 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:38.300108 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:38.296779 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:22:38.300108 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:38.297498 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:22:40.203853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:40.203587 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb"] Oct 30 18:22:40.213863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:40.213832 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9196a400-x77kb"] Oct 30 18:22:40.813613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:40.813574 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e4bee2a-07c4-4a0b-87ae-16cbd341de21" path="/var/lib/kubelet/pods/1e4bee2a-07c4-4a0b-87ae-16cbd341de21/volumes" Oct 30 18:22:41.554755 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:41.554722 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr"] Oct 30 18:22:41.555694 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:41.555659 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" podUID="5b66639f-9863-4681-b052-af4d1007c74b" containerName="istio-proxy" containerID="cri-o://2ce34298f678e487bfdec256893fdd00fc11a25626758ada2f037b01d858abd0" gracePeriod=30 Oct 30 18:22:41.959757 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:41.959678 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq" Oct 30 18:22:41.959757 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:41.959759 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg" Oct 30 18:22:46.595815 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.595788 5232 scope.go:122] "RemoveContainer" containerID="5e13dfe252a3e1ebbfb26e11681e7b8247616b0ee0fe1c03b5b687628ea40e5e" Oct 30 18:22:46.672457 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.672430 5232 scope.go:122] "RemoveContainer" containerID="77e7d2c66f0777149ec0e5524b0b273e25786f37983ef9c888dc8b48c6141d53" Oct 30 18:22:46.682924 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.682884 5232 scope.go:122] "RemoveContainer" containerID="715bd0736dd3b9a0b96094fc2db7ad9f4a7712b3b4886310eb0e230445af6680" Oct 30 18:22:46.696956 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.696929 5232 scope.go:122] "RemoveContainer" containerID="ff125f1132d1f0a41111489c74387db5b06d193bd3017cf63d81bc245aec9185" Oct 30 18:22:46.710346 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.710314 5232 scope.go:122] "RemoveContainer" containerID="eb1436ad530ca299796aa72cb7eed54770fcb2a48a72ab4a3c8fbd62788e4a5f" Oct 30 18:22:46.723776 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.723754 5232 scope.go:122] "RemoveContainer" containerID="cff94f5d33b62fc97d1ef856bebdc4dd54575acdceddd0d1ecca4ed6fc440cbb" Oct 30 18:22:46.735035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.735012 5232 scope.go:122] "RemoveContainer" containerID="602f6536b61e823920f912d64ae57f8afd13ba3cfb3b8063c827cd9d1f4b6965" Oct 30 18:22:46.754372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.754231 5232 scope.go:122] "RemoveContainer" containerID="6f1ab5924188988bfc26f587192f5cfd604155e5ae8e79c9d9f81aff76107724" Oct 30 18:22:46.877640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.877619 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:22:46.937075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.937037 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-workload-socket\") pod \"5b66639f-9863-4681-b052-af4d1007c74b\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " Oct 30 18:22:46.938612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.937403 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-workload-socket" pod "5b66639f-9863-4681-b052-af4d1007c74b" (UID: "5b66639f-9863-4681-b052-af4d1007c74b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:46.938612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.938005 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-istio-data" pod "5b66639f-9863-4681-b052-af4d1007c74b" (UID: "5b66639f-9863-4681-b052-af4d1007c74b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:46.938612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.938048 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-istio-data\") pod \"5b66639f-9863-4681-b052-af4d1007c74b\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " Oct 30 18:22:46.938612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.938177 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-workload-certs\") pod \"5b66639f-9863-4681-b052-af4d1007c74b\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " Oct 30 18:22:46.938612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.938206 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b66639f-9863-4681-b052-af4d1007c74b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b66639f-9863-4681-b052-af4d1007c74b-istiod-ca-cert\") pod \"5b66639f-9863-4681-b052-af4d1007c74b\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " Oct 30 18:22:46.938612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.938276 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b66639f-9863-4681-b052-af4d1007c74b-kube-api-access-g5rx7\" (UniqueName: \"kubernetes.io/projected/5b66639f-9863-4681-b052-af4d1007c74b-kube-api-access-g5rx7\") pod \"5b66639f-9863-4681-b052-af4d1007c74b\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " Oct 30 18:22:46.938612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.938330 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-istio-envoy\") pod \"5b66639f-9863-4681-b052-af4d1007c74b\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " Oct 30 18:22:46.938612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.938371 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b66639f-9863-4681-b052-af4d1007c74b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b66639f-9863-4681-b052-af4d1007c74b-istio-podinfo\") pod \"5b66639f-9863-4681-b052-af4d1007c74b\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " Oct 30 18:22:46.938612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.938542 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b66639f-9863-4681-b052-af4d1007c74b-istio-token\" (UniqueName: \"kubernetes.io/projected/5b66639f-9863-4681-b052-af4d1007c74b-istio-token\") pod \"5b66639f-9863-4681-b052-af4d1007c74b\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " Oct 30 18:22:46.938612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.938601 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-workload-certs" pod "5b66639f-9863-4681-b052-af4d1007c74b" (UID: "5b66639f-9863-4681-b052-af4d1007c74b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:46.939338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.938639 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-credential-socket\") pod \"5b66639f-9863-4681-b052-af4d1007c74b\" (UID: \"5b66639f-9863-4681-b052-af4d1007c74b\") " Oct 30 18:22:46.939338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.938764 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b66639f-9863-4681-b052-af4d1007c74b-istiod-ca-cert" pod "5b66639f-9863-4681-b052-af4d1007c74b" (UID: "5b66639f-9863-4681-b052-af4d1007c74b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:22:46.939669 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.939611 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-credential-socket" pod "5b66639f-9863-4681-b052-af4d1007c74b" (UID: "5b66639f-9863-4681-b052-af4d1007c74b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:46.940426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.940210 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.940426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.940235 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.940426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.940250 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.940426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.940261 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b66639f-9863-4681-b052-af4d1007c74b-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.940426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.940273 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:22:46.942804 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.942778 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-istio-envoy" pod "5b66639f-9863-4681-b052-af4d1007c74b" (UID: "5b66639f-9863-4681-b052-af4d1007c74b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:22:46.943330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.943304 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b66639f-9863-4681-b052-af4d1007c74b-istio-podinfo" pod "5b66639f-9863-4681-b052-af4d1007c74b" (UID: "5b66639f-9863-4681-b052-af4d1007c74b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:22:46.943774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.943751 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b66639f-9863-4681-b052-af4d1007c74b-kube-api-access-g5rx7" pod "5b66639f-9863-4681-b052-af4d1007c74b" (UID: "5b66639f-9863-4681-b052-af4d1007c74b"). InnerVolumeSpecName "kube-api-access-g5rx7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:46.944559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:46.944489 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b66639f-9863-4681-b052-af4d1007c74b-istio-token" pod "5b66639f-9863-4681-b052-af4d1007c74b" (UID: "5b66639f-9863-4681-b052-af4d1007c74b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:22:47.042223 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:47.042182 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g5rx7\" (UniqueName: \"kubernetes.io/projected/5b66639f-9863-4681-b052-af4d1007c74b-kube-api-access-g5rx7\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:22:47.042223 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:47.042215 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b66639f-9863-4681-b052-af4d1007c74b-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:22:47.042223 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:47.042224 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b66639f-9863-4681-b052-af4d1007c74b-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:22:47.042461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:47.042234 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b66639f-9863-4681-b052-af4d1007c74b-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:22:47.604475 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:47.604440 5232 generic.go:356] "Generic (PLEG): container finished" podID="5b66639f-9863-4681-b052-af4d1007c74b" containerID="2ce34298f678e487bfdec256893fdd00fc11a25626758ada2f037b01d858abd0" exitCode=0 Oct 30 18:22:47.604969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:47.604502 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" Oct 30 18:22:47.604969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:47.604538 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" event={"ID":"5b66639f-9863-4681-b052-af4d1007c74b","Type":"ContainerDied","Data":"2ce34298f678e487bfdec256893fdd00fc11a25626758ada2f037b01d858abd0"} Oct 30 18:22:47.604969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:47.604623 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr" event={"ID":"5b66639f-9863-4681-b052-af4d1007c74b","Type":"ContainerDied","Data":"e442b8c79bbbaf9af552b9bb0e36437376ba2f6ec17b7552823123c202950d5c"} Oct 30 18:22:47.604969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:47.604662 5232 scope.go:122] "RemoveContainer" containerID="2ce34298f678e487bfdec256893fdd00fc11a25626758ada2f037b01d858abd0" Oct 30 18:22:47.617329 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:47.617310 5232 scope.go:122] "RemoveContainer" containerID="2ce34298f678e487bfdec256893fdd00fc11a25626758ada2f037b01d858abd0" Oct 30 18:22:47.617679 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:22:47.617656 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ce34298f678e487bfdec256893fdd00fc11a25626758ada2f037b01d858abd0\": container with ID starting with 2ce34298f678e487bfdec256893fdd00fc11a25626758ada2f037b01d858abd0 not found: ID does not exist" containerID="2ce34298f678e487bfdec256893fdd00fc11a25626758ada2f037b01d858abd0" Oct 30 18:22:47.617726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:47.617689 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ce34298f678e487bfdec256893fdd00fc11a25626758ada2f037b01d858abd0"} err="failed to get container status \"2ce34298f678e487bfdec256893fdd00fc11a25626758ada2f037b01d858abd0\": rpc error: code = NotFound desc = could not find container \"2ce34298f678e487bfdec256893fdd00fc11a25626758ada2f037b01d858abd0\": container with ID starting with 2ce34298f678e487bfdec256893fdd00fc11a25626758ada2f037b01d858abd0 not found: ID does not exist" Oct 30 18:22:47.660983 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:47.660950 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr"] Oct 30 18:22:47.665779 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:47.665753 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-d29pr"] Oct 30 18:22:48.027794 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:48.027761 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz"] Oct 30 18:22:48.032322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:48.032298 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65adf21e-4hvjz"] Oct 30 18:22:48.805304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:48.805274 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b66639f-9863-4681-b052-af4d1007c74b" path="/var/lib/kubelet/pods/5b66639f-9863-4681-b052-af4d1007c74b/volumes" Oct 30 18:22:48.805813 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:48.805798 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="804c225a-2123-430a-a36d-cf349bf258e7" path="/var/lib/kubelet/pods/804c225a-2123-430a-a36d-cf349bf258e7/volumes" Oct 30 18:22:50.030485 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:50.030446 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2"] Oct 30 18:22:50.043838 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:50.043800 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3984aee6-kz7q2"] Oct 30 18:22:50.806159 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:50.806066 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4dd65e37-2961-45d9-ab25-7db48029ebbe" path="/var/lib/kubelet/pods/4dd65e37-2961-45d9-ab25-7db48029ebbe/volumes" Oct 30 18:22:53.026985 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:53.026951 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2"] Oct 30 18:22:53.042195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:53.042151 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e70f68c8-k5kg2"] Oct 30 18:22:54.671461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:54.671430 5232 generic.go:356] "Generic (PLEG): container finished" podID="f76e41be-f30a-4c44-90cc-ef0b08127828" containerID="d5d32e7f2f8775dcbbc28f3ca718e746c9e85675da59073fd3b28667b8b0b5e8" exitCode=1 Oct 30 18:22:54.671888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:54.671532 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c97f9884-dsjzk" event={"ID":"f76e41be-f30a-4c44-90cc-ef0b08127828","Type":"ContainerDied","Data":"d5d32e7f2f8775dcbbc28f3ca718e746c9e85675da59073fd3b28667b8b0b5e8"} Oct 30 18:22:54.672016 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:54.672002 5232 scope.go:122] "RemoveContainer" containerID="d5d32e7f2f8775dcbbc28f3ca718e746c9e85675da59073fd3b28667b8b0b5e8" Oct 30 18:22:54.672577 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:54.672564 5232 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:22:54.806307 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:54.806277 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e407db5-83a8-4496-843e-0974196bd4d7" path="/var/lib/kubelet/pods/6e407db5-83a8-4496-843e-0974196bd4d7/volumes" Oct 30 18:22:55.078450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:55.078425 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 18:22:55.681938 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:55.681904 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c97f9884-dsjzk" event={"ID":"f76e41be-f30a-4c44-90cc-ef0b08127828","Type":"ContainerStarted","Data":"6d40f668e435acf97f6a97b17e6f88669431cd68c6ee6eb5c0ac74c6de85029f"} Oct 30 18:22:55.682432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:22:55.682131 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68c97f9884-dsjzk" Oct 30 18:22:57.036047 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:22:57.036009 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:23:03.030524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:03.030483 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5"] Oct 30 18:23:03.042219 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:03.042192 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-65aa4380-jz8g5"] Oct 30 18:23:04.804680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:04.804645 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1157f8e6-65df-4a9e-9df2-d11a0eb1825e" path="/var/lib/kubelet/pods/1157f8e6-65df-4a9e-9df2-d11a0eb1825e/volumes" Oct 30 18:23:06.694277 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:06.694251 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68c97f9884-dsjzk" Oct 30 18:23:12.073440 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.073407 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8"] Oct 30 18:23:12.074223 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.074207 5232 cpu_manager.go:414] "containerMap: removing container" podUID="5b66639f-9863-4681-b052-af4d1007c74b" containerName="istio-proxy" Oct 30 18:23:12.074269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.074225 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b66639f-9863-4681-b052-af4d1007c74b" containerName="istio-proxy" Oct 30 18:23:12.074367 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.074358 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b66639f-9863-4681-b052-af4d1007c74b" containerName="istio-proxy" Oct 30 18:23:12.079896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.079876 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.120622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.120586 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8"] Oct 30 18:23:12.129200 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.128872 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6"] Oct 30 18:23:12.141831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.141802 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.142167 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.142141 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6"] Oct 30 18:23:12.184632 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.184598 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.184803 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.184642 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee56829-eea8-4aa6-b006-1bb43b731693-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.184803 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.184776 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.184910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.184804 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.184910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.184834 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.185015 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.184992 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee56829-eea8-4aa6-b006-1bb43b731693-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.185103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.185091 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.185147 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.185133 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.185194 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.185157 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.185250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.185227 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.185291 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.185257 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgddb\" (UniqueName: \"kubernetes.io/projected/cee56829-eea8-4aa6-b006-1bb43b731693-kube-api-access-cgddb\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.185291 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.185273 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.185401 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.185332 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grt9q\" (UniqueName: \"kubernetes.io/projected/6727049d-37d3-47f0-bf07-d63b6bf9aab4-kube-api-access-grt9q\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.185459 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.185435 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.185536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.185461 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee56829-eea8-4aa6-b006-1bb43b731693-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.185612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.185592 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.185660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.185650 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.185752 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.185738 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.286630 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.286602 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgddb\" (UniqueName: \"kubernetes.io/projected/cee56829-eea8-4aa6-b006-1bb43b731693-kube-api-access-cgddb\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.286630 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.286632 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.286809 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.286660 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grt9q\" (UniqueName: \"kubernetes.io/projected/6727049d-37d3-47f0-bf07-d63b6bf9aab4-kube-api-access-grt9q\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.286809 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.286780 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.286809 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.286806 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee56829-eea8-4aa6-b006-1bb43b731693-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.286930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.286858 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.286930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.286904 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.287029 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.286942 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.287029 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.287003 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.287029 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.287026 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee56829-eea8-4aa6-b006-1bb43b731693-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.287160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.287096 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.287160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.287122 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.287160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.287150 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.287319 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.287186 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee56829-eea8-4aa6-b006-1bb43b731693-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.287319 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.287275 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.287432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.287343 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.287432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.287382 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.287432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.287404 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.287663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.287448 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.287663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.287461 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.287663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.287494 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.287663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.287185 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-istio-data\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.287888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.287699 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.288034 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.287959 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-credential-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.288034 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.288013 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-workload-certs\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.288034 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.288026 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-workload-socket\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.288385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.288256 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee56829-eea8-4aa6-b006-1bb43b731693-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.288828 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.288804 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istiod-ca-cert\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.290085 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.290062 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.290183 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.290080 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.290183 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.290068 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-envoy\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.290183 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.290174 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee56829-eea8-4aa6-b006-1bb43b731693-istio-podinfo\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.305652 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.305626 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgddb\" (UniqueName: \"kubernetes.io/projected/cee56829-eea8-4aa6-b006-1bb43b731693-kube-api-access-cgddb\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.305761 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.305632 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.306290 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.306265 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee56829-eea8-4aa6-b006-1bb43b731693-istio-token\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.306419 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.306403 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grt9q\" (UniqueName: \"kubernetes.io/projected/6727049d-37d3-47f0-bf07-d63b6bf9aab4-kube-api-access-grt9q\") pod \"e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.396517 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.396444 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:12.457105 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.457041 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:12.717112 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.717084 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8"] Oct 30 18:23:12.718253 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:23:12.718227 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6727049d_37d3_47f0_bf07_d63b6bf9aab4.slice/crio-3b6c42222f8b8cfa5e930280cf8eed82dee1441ebb83dcf7be76eaafbf157d24 WatchSource:0}: Error finding container 3b6c42222f8b8cfa5e930280cf8eed82dee1441ebb83dcf7be76eaafbf157d24: Status 404 returned error can't find the container with id 3b6c42222f8b8cfa5e930280cf8eed82dee1441ebb83dcf7be76eaafbf157d24 Oct 30 18:23:12.721095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.721061 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:23:12.721189 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.721128 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:23:12.721189 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.721156 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:23:12.791342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.791314 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6"] Oct 30 18:23:12.792638 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:23:12.792616 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcee56829_eea8_4aa6_b006_1bb43b731693.slice/crio-170d08981f8d4e161d1e0f61ca85dba5346fe35dedb5bf6bb3faffb89b3230c0 WatchSource:0}: Error finding container 170d08981f8d4e161d1e0f61ca85dba5346fe35dedb5bf6bb3faffb89b3230c0: Status 404 returned error can't find the container with id 170d08981f8d4e161d1e0f61ca85dba5346fe35dedb5bf6bb3faffb89b3230c0 Oct 30 18:23:12.795769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.795730 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:23:12.795887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.795822 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:23:12.795887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.795861 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:23:12.852195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.852141 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" event={"ID":"6727049d-37d3-47f0-bf07-d63b6bf9aab4","Type":"ContainerStarted","Data":"2fc65f582b1aa534c940310ef99259fcbdf80cbf2e01a21b208668f05f3c5855"} Oct 30 18:23:12.852564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.852501 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" event={"ID":"6727049d-37d3-47f0-bf07-d63b6bf9aab4","Type":"ContainerStarted","Data":"3b6c42222f8b8cfa5e930280cf8eed82dee1441ebb83dcf7be76eaafbf157d24"} Oct 30 18:23:12.854039 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.854018 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" event={"ID":"cee56829-eea8-4aa6-b006-1bb43b731693","Type":"ContainerStarted","Data":"170d08981f8d4e161d1e0f61ca85dba5346fe35dedb5bf6bb3faffb89b3230c0"} Oct 30 18:23:12.896531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:12.896430 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podStartSLOduration=0.89641181 podStartE2EDuration="896.41181ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:12.892060702 +0000 UTC m=+1292.208481025" watchObservedRunningTime="2026-10-30 18:23:12.89641181 +0000 UTC m=+1292.212832132" Oct 30 18:23:13.397265 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:13.397226 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:13.398790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:13.398767 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:13.398853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:13.398829 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:13.865119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:13.865072 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" event={"ID":"cee56829-eea8-4aa6-b006-1bb43b731693","Type":"ContainerStarted","Data":"61b7dcaa069062171dc1b3a45fd7d0ecc9b6d0e9f0a6d1c59300f3271f14dd1c"} Oct 30 18:23:13.937288 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:13.937219 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" podStartSLOduration=1.937200077 podStartE2EDuration="1.937200077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:13.931737138 +0000 UTC m=+1293.248157462" watchObservedRunningTime="2026-10-30 18:23:13.937200077 +0000 UTC m=+1293.253620399" Oct 30 18:23:14.396946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:14.396917 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:14.397183 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:14.397154 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:14.457992 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:14.457951 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:14.460019 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:14.459991 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 18:23:14.460212 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:14.460054 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" podUID="cee56829-eea8-4aa6-b006-1bb43b731693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 18:23:15.396867 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:15.396837 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:15.397069 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:15.396965 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:15.458192 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:15.458157 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 18:23:15.458712 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:15.458209 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" podUID="cee56829-eea8-4aa6-b006-1bb43b731693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 18:23:16.042941 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:16.042906 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f"] Oct 30 18:23:16.047873 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:16.047795 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aee7087a-kkz2f"] Oct 30 18:23:16.397002 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:16.396911 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:16.397002 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:16.396983 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:16.457784 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:16.457747 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 18:23:16.457953 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:16.457806 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" podUID="cee56829-eea8-4aa6-b006-1bb43b731693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 18:23:16.805786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:16.805753 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d08e350-a49a-444e-b60b-0d1b97680381" path="/var/lib/kubelet/pods/1d08e350-a49a-444e-b60b-0d1b97680381/volumes" Oct 30 18:23:17.039207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:17.039168 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8"] Oct 30 18:23:17.043974 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:17.043943 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-39227f26-7k4v8"] Oct 30 18:23:17.397283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:17.397251 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:17.397468 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:17.397309 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:17.458431 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:17.458398 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 18:23:17.458641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:17.458461 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" podUID="cee56829-eea8-4aa6-b006-1bb43b731693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 18:23:18.398038 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:18.397886 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:18.398038 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:18.397958 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:18.464272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:18.464240 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" start-of-body= Oct 30 18:23:18.464489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:18.464299 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" podUID="cee56829-eea8-4aa6-b006-1bb43b731693" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.163:15021/healthz/ready\": dial tcp 10.133.0.163:15021: connect: connection refused" Oct 30 18:23:18.805123 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:18.805093 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddfc42c9-0e07-4789-8197-e70ca31c4769" path="/var/lib/kubelet/pods/ddfc42c9-0e07-4789-8197-e70ca31c4769/volumes" Oct 30 18:23:19.030486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:19.030455 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p"] Oct 30 18:23:19.066010 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:19.065917 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-516db80f-ghd9p"] Oct 30 18:23:19.397284 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:19.397207 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:19.397284 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:19.397259 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:19.462473 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:19.462446 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:19.462949 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:19.462931 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:19.463785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:19.463770 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:23:20.397745 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:20.397706 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:20.397970 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:20.397768 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:20.820640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:20.820587 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a13de057-8d57-4792-a002-990d43df53d5" path="/var/lib/kubelet/pods/a13de057-8d57-4792-a002-990d43df53d5/volumes" Oct 30 18:23:21.397475 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:21.397436 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:21.397728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:21.397502 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:22.396944 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:22.396912 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:22.397370 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:22.396972 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:23.397449 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:23.397411 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:23.397878 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:23.397479 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:24.397420 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:24.397384 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:24.397653 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:24.397437 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:25.397447 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:25.397411 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:25.397630 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:25.397473 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:26.397489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:26.397449 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:26.397920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:26.397524 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:26.649657 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:26.649570 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw"] Oct 30 18:23:26.662189 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:26.662165 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw" Oct 30 18:23:26.662680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:26.662204 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw"] Oct 30 18:23:26.677406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:26.677379 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-295497\"" Oct 30 18:23:26.689397 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:26.689375 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-295497-dockercfg-6hmv9\"" Oct 30 18:23:26.851177 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:26.851146 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ee5406-0fa7-4e83-a199-9fc329adba3b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw\" (UID: \"06ee5406-0fa7-4e83-a199-9fc329adba3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw" Oct 30 18:23:26.851904 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:26.851842 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc9v9\" (UniqueName: \"kubernetes.io/projected/06ee5406-0fa7-4e83-a199-9fc329adba3b-kube-api-access-vc9v9\") pod \"payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw\" (UID: \"06ee5406-0fa7-4e83-a199-9fc329adba3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw" Oct 30 18:23:26.953465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:26.953371 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc9v9\" (UniqueName: \"kubernetes.io/projected/06ee5406-0fa7-4e83-a199-9fc329adba3b-kube-api-access-vc9v9\") pod \"payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw\" (UID: \"06ee5406-0fa7-4e83-a199-9fc329adba3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw" Oct 30 18:23:26.953667 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:26.953481 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ee5406-0fa7-4e83-a199-9fc329adba3b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw\" (UID: \"06ee5406-0fa7-4e83-a199-9fc329adba3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw" Oct 30 18:23:26.954143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:26.954123 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ee5406-0fa7-4e83-a199-9fc329adba3b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw\" (UID: \"06ee5406-0fa7-4e83-a199-9fc329adba3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw" Oct 30 18:23:26.972180 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:26.972154 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc9v9\" (UniqueName: \"kubernetes.io/projected/06ee5406-0fa7-4e83-a199-9fc329adba3b-kube-api-access-vc9v9\") pod \"payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw\" (UID: \"06ee5406-0fa7-4e83-a199-9fc329adba3b\") " pod="openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw" Oct 30 18:23:26.980916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:26.980892 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw" Oct 30 18:23:27.035878 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:27.035843 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx"] Oct 30 18:23:27.038031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:27.037997 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b76989c0-pvmxx"] Oct 30 18:23:27.045410 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:23:27.045368 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:23:27.290056 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:27.290021 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw"] Oct 30 18:23:27.291459 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:23:27.291427 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ee5406_0fa7_4e83_a199_9fc329adba3b.slice/crio-a4f286c6ce34f6e9b9f2e2079e28d58ae8b68f160dc2f8053fb550493c0e1d41 WatchSource:0}: Error finding container a4f286c6ce34f6e9b9f2e2079e28d58ae8b68f160dc2f8053fb550493c0e1d41: Status 404 returned error can't find the container with id a4f286c6ce34f6e9b9f2e2079e28d58ae8b68f160dc2f8053fb550493c0e1d41 Oct 30 18:23:27.396588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:27.396559 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:27.397061 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:27.396840 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:27.397061 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:27.396886 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:28.046790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:28.046752 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw" event={"ID":"06ee5406-0fa7-4e83-a199-9fc329adba3b","Type":"ContainerStarted","Data":"6c3ac1908aaf13cde5ba9b289e97fafa866e7d697e64daf1ece0ab562b52c62a"} Oct 30 18:23:28.047203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:28.046810 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw" event={"ID":"06ee5406-0fa7-4e83-a199-9fc329adba3b","Type":"ContainerStarted","Data":"a4f286c6ce34f6e9b9f2e2079e28d58ae8b68f160dc2f8053fb550493c0e1d41"} Oct 30 18:23:28.047203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:28.046839 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw" Oct 30 18:23:28.086903 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:28.086853 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw" podStartSLOduration=2.086840195 podStartE2EDuration="2.086840195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:28.083977186 +0000 UTC m=+1307.400397507" watchObservedRunningTime="2026-10-30 18:23:28.086840195 +0000 UTC m=+1307.403260516" Oct 30 18:23:28.418771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:28.418678 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:28.419034 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:28.419003 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:28.804958 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:28.804925 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ddb2fb1d-093e-420b-9b0c-3fe66306155d" path="/var/lib/kubelet/pods/ddb2fb1d-093e-420b-9b0c-3fe66306155d/volumes" Oct 30 18:23:29.494131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:29.494047 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:29.494614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:29.494582 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:30.400533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:30.400136 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:30.400533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:30.400223 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:31.396963 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:31.396926 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:31.397599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:31.397566 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:32.357533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:32.356610 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:32.411586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:32.410783 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:32.411586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:32.410844 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:32.516088 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:32.514906 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:32.516088 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:32.515694 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:23:32.592626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:32.585810 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:32.674591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:32.673887 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:23:32.833744 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:32.831821 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:23:32.864337 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:32.861764 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:23:32.991601 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:32.991556 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:33.103593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.099633 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:33.359593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.358618 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 18:23:33.359593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.359221 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:23:33.397076 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.397032 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:33.397364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.397327 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:33.517542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.516873 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:33.544533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.537153 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:23:33.552626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.552572 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:23:33.602534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.553614 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:23:33.683583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.674919 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:23:33.683583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.675876 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:33.683583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.683108 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:23:33.694341 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.684226 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:23:33.694341 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.685132 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:23:33.694341 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.685199 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:23:33.778638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.768534 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:23:33.778638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.770622 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:23:33.801395 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.801364 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:23:33.802041 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:33.802018 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:23:34.398334 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:34.398304 5232 patch_prober.go:29] interesting pod/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" start-of-body= Oct 30 18:23:34.398630 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:34.398362 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.162:15021/healthz/ready\": dial tcp 10.133.0.162:15021: connect: connection refused" Oct 30 18:23:35.402008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:35.401981 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:35.403169 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:35.403147 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:23:35.695089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:35.695020 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7"] Oct 30 18:23:35.700935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:35.700877 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7" Oct 30 18:23:35.717972 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:35.717942 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:23:35.718775 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:35.718751 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:23:35.721737 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:35.721709 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7"] Oct 30 18:23:35.791828 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:35.791796 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7\" (UID: \"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7" Oct 30 18:23:35.792025 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:35.791920 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7\" (UID: \"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7" Oct 30 18:23:35.792177 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:35.792163 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxwbw\" (UniqueName: \"kubernetes.io/projected/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-kube-api-access-mxwbw\") pod \"maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7\" (UID: \"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7" Oct 30 18:23:35.893607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:35.893573 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxwbw\" (UniqueName: \"kubernetes.io/projected/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-kube-api-access-mxwbw\") pod \"maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7\" (UID: \"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7" Oct 30 18:23:35.893826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:35.893745 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7\" (UID: \"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7" Oct 30 18:23:35.893922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:35.893891 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7\" (UID: \"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7" Oct 30 18:23:35.894550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:35.894483 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7\" (UID: \"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7" Oct 30 18:23:35.896299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:35.896279 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7\" (UID: \"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7" Oct 30 18:23:35.915224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:35.915199 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxwbw\" (UniqueName: \"kubernetes.io/projected/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-kube-api-access-mxwbw\") pod \"maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7\" (UID: \"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7" Oct 30 18:23:36.031287 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:36.031258 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7" Oct 30 18:23:36.339604 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:36.339569 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7"] Oct 30 18:23:36.341255 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:23:36.341230 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d8e3953_b0b1_4570_89bb_c76e2f3f0a0c.slice/crio-b1e926a9b0203d985b7fea54579c95cff82ae9db5a50c1fe03b195bd47cabb62 WatchSource:0}: Error finding container b1e926a9b0203d985b7fea54579c95cff82ae9db5a50c1fe03b195bd47cabb62: Status 404 returned error can't find the container with id b1e926a9b0203d985b7fea54579c95cff82ae9db5a50c1fe03b195bd47cabb62 Oct 30 18:23:37.311018 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:37.310975 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7" event={"ID":"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c","Type":"ContainerStarted","Data":"08dfaf70bcfe59935079d5ba74e77a5e41ed0ecb2a97d26de789f84f1c4469df"} Oct 30 18:23:37.311393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:37.311043 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7" event={"ID":"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c","Type":"ContainerStarted","Data":"b1e926a9b0203d985b7fea54579c95cff82ae9db5a50c1fe03b195bd47cabb62"} Oct 30 18:23:37.350374 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:37.350331 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7" podStartSLOduration=2.350317604 podStartE2EDuration="2.350317604s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:37.350160861 +0000 UTC m=+1316.666581182" watchObservedRunningTime="2026-10-30 18:23:37.350317604 +0000 UTC m=+1316.666737924" Oct 30 18:23:38.368533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:38.365627 5232 generic.go:356] "Generic (PLEG): container finished" podID="5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c" containerID="08dfaf70bcfe59935079d5ba74e77a5e41ed0ecb2a97d26de789f84f1c4469df" exitCode=0 Oct 30 18:23:38.368533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:38.365809 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7" event={"ID":"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c","Type":"ContainerDied","Data":"08dfaf70bcfe59935079d5ba74e77a5e41ed0ecb2a97d26de789f84f1c4469df"} Oct 30 18:23:39.063902 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:39.063876 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw" Oct 30 18:23:39.631875 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:39.631848 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7" Oct 30 18:23:39.744980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:39.744942 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-maas-api-xks-ca\") pod \"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c\" (UID: \"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c\") " Oct 30 18:23:39.745173 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:39.745041 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-kube-api-access-mxwbw\" (UniqueName: \"kubernetes.io/projected/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-kube-api-access-mxwbw\") pod \"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c\" (UID: \"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c\") " Oct 30 18:23:39.745173 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:39.745146 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-maas-api-openshift-ca\") pod \"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c\" (UID: \"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c\") " Oct 30 18:23:39.745601 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:39.745571 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-maas-api-openshift-ca" pod "5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c" (UID: "5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:39.746074 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:39.746056 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:39.747350 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:39.747320 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-maas-api-xks-ca" pod "5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c" (UID: "5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:23:39.747431 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:39.747356 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-kube-api-access-mxwbw" pod "5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c" (UID: "5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c"). InnerVolumeSpecName "kube-api-access-mxwbw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:39.847941 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:39.847907 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxwbw\" (UniqueName: \"kubernetes.io/projected/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-kube-api-access-mxwbw\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:39.847941 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:39.847941 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:40.594538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.590671 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk"] Oct 30 18:23:40.594538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.590970 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" podUID="17ccda47-32b1-42f6-9030-0c040fc73883" containerName="istio-proxy" containerID="cri-o://4762ad491b3bde8ce19235625990fe0e46e2ab599ccc9dbc6bb619b096b441d2" gracePeriod=30 Oct 30 18:23:40.611581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.609204 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw"] Oct 30 18:23:40.611581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.609451 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" podUID="b090d280-3716-4098-9330-7fd924785b9d" containerName="istio-proxy" containerID="cri-o://4f27afc9fb40171ec21b0c5657cf3d798240d87e1fcf0621a1b6ce9556682cb5" gracePeriod=30 Oct 30 18:23:40.625528 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.617364 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7" event={"ID":"5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c","Type":"ContainerDied","Data":"b1e926a9b0203d985b7fea54579c95cff82ae9db5a50c1fe03b195bd47cabb62"} Oct 30 18:23:40.625528 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.617425 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b1e926a9b0203d985b7fea54579c95cff82ae9db5a50c1fe03b195bd47cabb62" Oct 30 18:23:40.625528 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.617653 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7" Oct 30 18:23:40.632662 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.630261 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527"] Oct 30 18:23:40.632662 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.630488 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" podUID="c5923f01-3342-409d-8cc6-49e03be3a4ab" containerName="istio-proxy" containerID="cri-o://2061a643b81527aa1f3829aba1ef6a731b19d1013c718e825d1f7a482d4c0cbf" gracePeriod=30 Oct 30 18:23:40.639438 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.639180 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6"] Oct 30 18:23:40.640043 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.639996 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" podUID="897d6674-2313-4b21-8a84-d84d56d1c22e" containerName="istio-proxy" containerID="cri-o://88b8a3bed368fe3fc9a323daf6ab1b1babcc75f691270e1c60b49f2c8155e453" gracePeriod=30 Oct 30 18:23:40.646270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.646249 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2"] Oct 30 18:23:40.646429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.646405 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" podUID="6c7a6701-721d-448e-af9a-10216921bf48" containerName="istio-proxy" containerID="cri-o://8eb2dc9ae29604190a5d4a178668d778f78580546111260f2e2e59af72d54162" gracePeriod=30 Oct 30 18:23:40.653096 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.653070 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj"] Oct 30 18:23:40.653279 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.653259 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" podUID="94b06967-baa7-4770-9632-f30f27f1dcae" containerName="istio-proxy" containerID="cri-o://f0c3ce9cefec72aa003aad71bbf47d3f013da4f47c46676a85da9c7380cc124c" gracePeriod=30 Oct 30 18:23:40.661191 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.660865 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974"] Oct 30 18:23:40.661191 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.661043 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" podUID="f8c0e348-2b8d-48e2-9ec0-a21a83db391e" containerName="istio-proxy" containerID="cri-o://f15ae61695576b22e3376adff368e02d9edb1454d65fd6cc186ba20f5147687d" gracePeriod=30 Oct 30 18:23:40.680419 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.680385 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6"] Oct 30 18:23:40.683730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.683662 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" podUID="0bbd2e9b-4989-46ca-bf9b-afe2ef671726" containerName="istio-proxy" containerID="cri-o://660df6d969acdecc8c744f0e7aab5c6755bb3717fede45654cc9261893ebcbf0" gracePeriod=30 Oct 30 18:23:40.686911 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.686894 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5"] Oct 30 18:23:40.687262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.687239 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" podUID="f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" containerName="istio-proxy" containerID="cri-o://7e52361b9c056affff55f26709a0ea054c70cd21d17787aa1e947e64eb17c278" gracePeriod=30 Oct 30 18:23:40.692965 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.692942 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247"] Oct 30 18:23:40.693195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:40.693174 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" podUID="31f08f06-6d9f-4003-b35a-62de857fbe80" containerName="istio-proxy" containerID="cri-o://ce86ab799b5b32e654b04d184f606cafb1c99b890e0d9558fe170b221ea81793" gracePeriod=30 Oct 30 18:23:44.349962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.349929 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2"] Oct 30 18:23:44.350993 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.350968 5232 cpu_manager.go:414] "containerMap: removing container" podUID="5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c" containerName="revoke-keys" Oct 30 18:23:44.350993 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.350992 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c" containerName="revoke-keys" Oct 30 18:23:44.351279 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.351263 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c" containerName="revoke-keys" Oct 30 18:23:44.356647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.356608 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.381070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.381035 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2"] Oct 30 18:23:44.385545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.385490 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-9066aa-openshift-default-dockercfg-4s6lw\"" Oct 30 18:23:44.476546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.474894 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-workload-certs\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.476546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.474954 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-podinfo\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.476546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.475007 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-envoy\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.476546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.475221 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istiod-ca-cert\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.476546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.475356 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-96dtv\" (UniqueName: \"kubernetes.io/projected/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-kube-api-access-96dtv\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.476546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.475493 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-token\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.476546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.475567 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-data\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.476546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.475598 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-credential-socket\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.476546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.475751 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-workload-socket\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.591540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.590120 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-envoy\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.591540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.590336 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istiod-ca-cert\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.591540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.590448 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-96dtv\" (UniqueName: \"kubernetes.io/projected/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-kube-api-access-96dtv\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.591540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.590591 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-token\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.591540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.590632 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-data\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.591540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.590662 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-credential-socket\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.591540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.590736 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-workload-socket\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.591540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.590817 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-workload-certs\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.591540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.590848 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-podinfo\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.643588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.613192 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-podinfo\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.643588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.613889 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-data\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.643588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.614137 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-credential-socket\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.643588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.614396 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-workload-socket\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.643588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.614656 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-workload-certs\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.643588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.615256 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istiod-ca-cert\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.715538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.715373 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-envoy\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.726529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.716127 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-token\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.726529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.720123 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-96dtv\" (UniqueName: \"kubernetes.io/projected/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-kube-api-access-96dtv\") pod \"e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:44.972974 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:44.972893 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:45.474137 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:45.474104 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2"] Oct 30 18:23:45.475205 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:23:45.475175 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dc14e97_fdc1_4a4c_9c76_f0f63cbb54b6.slice/crio-244e624eb0e1d066097c550cff1f49f01e236b7aeebaeedf72faed216f5bce23 WatchSource:0}: Error finding container 244e624eb0e1d066097c550cff1f49f01e236b7aeebaeedf72faed216f5bce23: Status 404 returned error can't find the container with id 244e624eb0e1d066097c550cff1f49f01e236b7aeebaeedf72faed216f5bce23 Oct 30 18:23:45.478030 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:45.477996 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:23:45.478095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:45.478064 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:23:45.478137 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:45.478098 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:23:45.817635 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:45.817605 5232 generic.go:356] "Generic (PLEG): container finished" podID="17ccda47-32b1-42f6-9030-0c040fc73883" containerID="4762ad491b3bde8ce19235625990fe0e46e2ab599ccc9dbc6bb619b096b441d2" exitCode=0 Oct 30 18:23:45.817924 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:45.817904 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" event={"ID":"17ccda47-32b1-42f6-9030-0c040fc73883","Type":"ContainerDied","Data":"4762ad491b3bde8ce19235625990fe0e46e2ab599ccc9dbc6bb619b096b441d2"} Oct 30 18:23:45.820049 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:45.820027 5232 generic.go:356] "Generic (PLEG): container finished" podID="6c7a6701-721d-448e-af9a-10216921bf48" containerID="8eb2dc9ae29604190a5d4a178668d778f78580546111260f2e2e59af72d54162" exitCode=0 Oct 30 18:23:45.820251 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:45.820234 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" event={"ID":"6c7a6701-721d-448e-af9a-10216921bf48","Type":"ContainerDied","Data":"8eb2dc9ae29604190a5d4a178668d778f78580546111260f2e2e59af72d54162"} Oct 30 18:23:45.822193 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:45.822172 5232 generic.go:356] "Generic (PLEG): container finished" podID="b090d280-3716-4098-9330-7fd924785b9d" containerID="4f27afc9fb40171ec21b0c5657cf3d798240d87e1fcf0621a1b6ce9556682cb5" exitCode=0 Oct 30 18:23:45.822447 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:45.822430 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" event={"ID":"b090d280-3716-4098-9330-7fd924785b9d","Type":"ContainerDied","Data":"4f27afc9fb40171ec21b0c5657cf3d798240d87e1fcf0621a1b6ce9556682cb5"} Oct 30 18:23:45.831445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:45.827835 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" event={"ID":"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6","Type":"ContainerStarted","Data":"760b4a2fa1a6e8045933313a9b9c2113acbddd616d424c879f758c55680d476e"} Oct 30 18:23:45.831445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:45.827898 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" event={"ID":"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6","Type":"ContainerStarted","Data":"244e624eb0e1d066097c550cff1f49f01e236b7aeebaeedf72faed216f5bce23"} Oct 30 18:23:45.901412 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:45.896322 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" podStartSLOduration=1.89630123 podStartE2EDuration="1.89630123s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:45.888811443 +0000 UTC m=+1325.205231766" watchObservedRunningTime="2026-10-30 18:23:45.89630123 +0000 UTC m=+1325.212721549" Oct 30 18:23:45.974454 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:45.974361 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:46.201600 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.201484 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:23:46.327832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.327800 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:23:46.373747 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.373719 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:23:46.414683 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.414652 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-istio-data\") pod \"17ccda47-32b1-42f6-9030-0c040fc73883\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " Oct 30 18:23:46.415002 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.414967 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-istio-data" pod "17ccda47-32b1-42f6-9030-0c040fc73883" (UID: "17ccda47-32b1-42f6-9030-0c040fc73883"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.415158 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.415135 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-credential-socket\") pod \"17ccda47-32b1-42f6-9030-0c040fc73883\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " Oct 30 18:23:46.415364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.415239 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-credential-socket" pod "17ccda47-32b1-42f6-9030-0c040fc73883" (UID: "17ccda47-32b1-42f6-9030-0c040fc73883"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.415564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.415357 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-istio-envoy\") pod \"17ccda47-32b1-42f6-9030-0c040fc73883\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " Oct 30 18:23:46.415904 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.415884 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-workload-certs\") pod \"17ccda47-32b1-42f6-9030-0c040fc73883\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " Oct 30 18:23:46.416154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.416134 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17ccda47-32b1-42f6-9030-0c040fc73883-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ccda47-32b1-42f6-9030-0c040fc73883-istio-podinfo\") pod \"17ccda47-32b1-42f6-9030-0c040fc73883\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " Oct 30 18:23:46.416292 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.416278 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ccda47-32b1-42f6-9030-0c040fc73883-istio-token\" (UniqueName: \"kubernetes.io/projected/17ccda47-32b1-42f6-9030-0c040fc73883-istio-token\") pod \"17ccda47-32b1-42f6-9030-0c040fc73883\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " Oct 30 18:23:46.416403 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.416390 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c7a6701-721d-448e-af9a-10216921bf48-kube-api-access-mrk6k\" (UniqueName: \"kubernetes.io/projected/6c7a6701-721d-448e-af9a-10216921bf48-kube-api-access-mrk6k\") pod \"6c7a6701-721d-448e-af9a-10216921bf48\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " Oct 30 18:23:46.416896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.416295 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-workload-certs" pod "17ccda47-32b1-42f6-9030-0c040fc73883" (UID: "17ccda47-32b1-42f6-9030-0c040fc73883"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.416896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.416567 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c7a6701-721d-448e-af9a-10216921bf48-istio-token\" (UniqueName: \"kubernetes.io/projected/6c7a6701-721d-448e-af9a-10216921bf48-istio-token\") pod \"6c7a6701-721d-448e-af9a-10216921bf48\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " Oct 30 18:23:46.416896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.416667 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-workload-certs\") pod \"b090d280-3716-4098-9330-7fd924785b9d\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " Oct 30 18:23:46.416896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.416693 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17ccda47-32b1-42f6-9030-0c040fc73883-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ccda47-32b1-42f6-9030-0c040fc73883-istiod-ca-cert\") pod \"17ccda47-32b1-42f6-9030-0c040fc73883\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " Oct 30 18:23:46.416896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.416754 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-istio-envoy\") pod \"6c7a6701-721d-448e-af9a-10216921bf48\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " Oct 30 18:23:46.416896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.416830 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17ccda47-32b1-42f6-9030-0c040fc73883-kube-api-access-jwbgk\" (UniqueName: \"kubernetes.io/projected/17ccda47-32b1-42f6-9030-0c040fc73883-kube-api-access-jwbgk\") pod \"17ccda47-32b1-42f6-9030-0c040fc73883\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " Oct 30 18:23:46.416896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.416864 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-workload-socket\") pod \"17ccda47-32b1-42f6-9030-0c040fc73883\" (UID: \"17ccda47-32b1-42f6-9030-0c040fc73883\") " Oct 30 18:23:46.417914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.417892 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17ccda47-32b1-42f6-9030-0c040fc73883-istiod-ca-cert" pod "17ccda47-32b1-42f6-9030-0c040fc73883" (UID: "17ccda47-32b1-42f6-9030-0c040fc73883"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:46.418139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.418112 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.418225 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.418147 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.418225 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.418161 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.418225 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.418188 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-istio-envoy" pod "17ccda47-32b1-42f6-9030-0c040fc73883" (UID: "17ccda47-32b1-42f6-9030-0c040fc73883"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.418810 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.418785 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-workload-socket" pod "17ccda47-32b1-42f6-9030-0c040fc73883" (UID: "17ccda47-32b1-42f6-9030-0c040fc73883"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.418914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.418819 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-workload-certs" pod "b090d280-3716-4098-9330-7fd924785b9d" (UID: "b090d280-3716-4098-9330-7fd924785b9d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.419843 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.419806 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7a6701-721d-448e-af9a-10216921bf48-kube-api-access-mrk6k" pod "6c7a6701-721d-448e-af9a-10216921bf48" (UID: "6c7a6701-721d-448e-af9a-10216921bf48"). InnerVolumeSpecName "kube-api-access-mrk6k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:46.421379 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.421362 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c7a6701-721d-448e-af9a-10216921bf48-istio-token" pod "6c7a6701-721d-448e-af9a-10216921bf48" (UID: "6c7a6701-721d-448e-af9a-10216921bf48"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:46.422979 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.422935 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-istio-envoy" pod "6c7a6701-721d-448e-af9a-10216921bf48" (UID: "6c7a6701-721d-448e-af9a-10216921bf48"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.422979 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.422958 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ccda47-32b1-42f6-9030-0c040fc73883-kube-api-access-jwbgk" pod "17ccda47-32b1-42f6-9030-0c040fc73883" (UID: "17ccda47-32b1-42f6-9030-0c040fc73883"). InnerVolumeSpecName "kube-api-access-jwbgk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:46.425879 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.425846 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17ccda47-32b1-42f6-9030-0c040fc73883-istio-podinfo" pod "17ccda47-32b1-42f6-9030-0c040fc73883" (UID: "17ccda47-32b1-42f6-9030-0c040fc73883"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:23:46.426810 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.426790 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17ccda47-32b1-42f6-9030-0c040fc73883-istio-token" pod "17ccda47-32b1-42f6-9030-0c040fc73883" (UID: "17ccda47-32b1-42f6-9030-0c040fc73883"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:46.519471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.519407 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-credential-socket\") pod \"6c7a6701-721d-448e-af9a-10216921bf48\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " Oct 30 18:23:46.522787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.520208 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-credential-socket" pod "6c7a6701-721d-448e-af9a-10216921bf48" (UID: "6c7a6701-721d-448e-af9a-10216921bf48"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.522787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.520331 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c7a6701-721d-448e-af9a-10216921bf48-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c7a6701-721d-448e-af9a-10216921bf48-istiod-ca-cert\") pod \"6c7a6701-721d-448e-af9a-10216921bf48\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " Oct 30 18:23:46.522787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.520994 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c7a6701-721d-448e-af9a-10216921bf48-istiod-ca-cert" pod "6c7a6701-721d-448e-af9a-10216921bf48" (UID: "6c7a6701-721d-448e-af9a-10216921bf48"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:46.522787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.522011 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-istio-data\") pod \"b090d280-3716-4098-9330-7fd924785b9d\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " Oct 30 18:23:46.522787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.522052 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b090d280-3716-4098-9330-7fd924785b9d-istio-token\" (UniqueName: \"kubernetes.io/projected/b090d280-3716-4098-9330-7fd924785b9d-istio-token\") pod \"b090d280-3716-4098-9330-7fd924785b9d\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " Oct 30 18:23:46.522787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.522377 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-istio-data" pod "b090d280-3716-4098-9330-7fd924785b9d" (UID: "b090d280-3716-4098-9330-7fd924785b9d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.524368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.523326 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b090d280-3716-4098-9330-7fd924785b9d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b090d280-3716-4098-9330-7fd924785b9d-istio-podinfo\") pod \"b090d280-3716-4098-9330-7fd924785b9d\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " Oct 30 18:23:46.524368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.523442 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-istio-envoy\") pod \"b090d280-3716-4098-9330-7fd924785b9d\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " Oct 30 18:23:46.524368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.523477 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-istio-data\") pod \"6c7a6701-721d-448e-af9a-10216921bf48\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " Oct 30 18:23:46.524368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.523526 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b090d280-3716-4098-9330-7fd924785b9d-kube-api-access-xr7tv\" (UniqueName: \"kubernetes.io/projected/b090d280-3716-4098-9330-7fd924785b9d-kube-api-access-xr7tv\") pod \"b090d280-3716-4098-9330-7fd924785b9d\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " Oct 30 18:23:46.524368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.523584 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-credential-socket\") pod \"b090d280-3716-4098-9330-7fd924785b9d\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " Oct 30 18:23:46.524368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.523615 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b090d280-3716-4098-9330-7fd924785b9d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b090d280-3716-4098-9330-7fd924785b9d-istiod-ca-cert\") pod \"b090d280-3716-4098-9330-7fd924785b9d\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " Oct 30 18:23:46.524368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.523694 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-workload-socket\") pod \"b090d280-3716-4098-9330-7fd924785b9d\" (UID: \"b090d280-3716-4098-9330-7fd924785b9d\") " Oct 30 18:23:46.524368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.523734 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-workload-socket\") pod \"6c7a6701-721d-448e-af9a-10216921bf48\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " Oct 30 18:23:46.524368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.523762 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-workload-certs\") pod \"6c7a6701-721d-448e-af9a-10216921bf48\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " Oct 30 18:23:46.524368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.523804 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c7a6701-721d-448e-af9a-10216921bf48-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c7a6701-721d-448e-af9a-10216921bf48-istio-podinfo\") pod \"6c7a6701-721d-448e-af9a-10216921bf48\" (UID: \"6c7a6701-721d-448e-af9a-10216921bf48\") " Oct 30 18:23:46.524368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.524094 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-istio-data" pod "6c7a6701-721d-448e-af9a-10216921bf48" (UID: "6c7a6701-721d-448e-af9a-10216921bf48"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.527269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.526672 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.527269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.526699 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17ccda47-32b1-42f6-9030-0c040fc73883-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.527269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.526713 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.527269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.526728 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jwbgk\" (UniqueName: \"kubernetes.io/projected/17ccda47-32b1-42f6-9030-0c040fc73883-kube-api-access-jwbgk\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.527269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.526743 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.527269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.526757 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.527269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.526769 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.527269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.526781 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17ccda47-32b1-42f6-9030-0c040fc73883-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.527269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.526794 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.527269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.526806 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c7a6701-721d-448e-af9a-10216921bf48-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.527269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.526820 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17ccda47-32b1-42f6-9030-0c040fc73883-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.527269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.526833 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17ccda47-32b1-42f6-9030-0c040fc73883-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.527269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.526846 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mrk6k\" (UniqueName: \"kubernetes.io/projected/6c7a6701-721d-448e-af9a-10216921bf48-kube-api-access-mrk6k\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.527269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.526860 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c7a6701-721d-448e-af9a-10216921bf48-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.554906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.529867 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-workload-socket" pod "b090d280-3716-4098-9330-7fd924785b9d" (UID: "b090d280-3716-4098-9330-7fd924785b9d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.554906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.530053 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-credential-socket" pod "b090d280-3716-4098-9330-7fd924785b9d" (UID: "b090d280-3716-4098-9330-7fd924785b9d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.554906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.530308 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b090d280-3716-4098-9330-7fd924785b9d-istiod-ca-cert" pod "b090d280-3716-4098-9330-7fd924785b9d" (UID: "b090d280-3716-4098-9330-7fd924785b9d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:46.554906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.530591 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-workload-certs" pod "6c7a6701-721d-448e-af9a-10216921bf48" (UID: "6c7a6701-721d-448e-af9a-10216921bf48"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.554906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.530834 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-workload-socket" pod "6c7a6701-721d-448e-af9a-10216921bf48" (UID: "6c7a6701-721d-448e-af9a-10216921bf48"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.637133 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.626100 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b090d280-3716-4098-9330-7fd924785b9d-istio-token" pod "b090d280-3716-4098-9330-7fd924785b9d" (UID: "b090d280-3716-4098-9330-7fd924785b9d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:46.637133 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.636300 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:23:46.638831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.638720 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.638831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.638744 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b090d280-3716-4098-9330-7fd924785b9d-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.638831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.638754 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.638831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.638765 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.638831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.638773 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c7a6701-721d-448e-af9a-10216921bf48-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.638831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.638782 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b090d280-3716-4098-9330-7fd924785b9d-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.648319 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.639252 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b090d280-3716-4098-9330-7fd924785b9d-istio-podinfo" pod "b090d280-3716-4098-9330-7fd924785b9d" (UID: "b090d280-3716-4098-9330-7fd924785b9d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:23:46.648319 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.641989 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c7a6701-721d-448e-af9a-10216921bf48-istio-podinfo" pod "6c7a6701-721d-448e-af9a-10216921bf48" (UID: "6c7a6701-721d-448e-af9a-10216921bf48"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:23:46.648319 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.647760 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b090d280-3716-4098-9330-7fd924785b9d-kube-api-access-xr7tv" pod "b090d280-3716-4098-9330-7fd924785b9d" (UID: "b090d280-3716-4098-9330-7fd924785b9d"). InnerVolumeSpecName "kube-api-access-xr7tv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:46.648319 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.647874 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-istio-envoy" pod "b090d280-3716-4098-9330-7fd924785b9d" (UID: "b090d280-3716-4098-9330-7fd924785b9d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.822345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.782747 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:23:46.822345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.793328 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:23:46.822345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.793949 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31f08f06-6d9f-4003-b35a-62de857fbe80-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f08f06-6d9f-4003-b35a-62de857fbe80-istiod-ca-cert\") pod \"31f08f06-6d9f-4003-b35a-62de857fbe80\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " Oct 30 18:23:46.822345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.793997 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-data\") pod \"31f08f06-6d9f-4003-b35a-62de857fbe80\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " Oct 30 18:23:46.822345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.794091 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-envoy\") pod \"31f08f06-6d9f-4003-b35a-62de857fbe80\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " Oct 30 18:23:46.822345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.794274 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-token\" (UniqueName: \"kubernetes.io/projected/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-token\") pod \"31f08f06-6d9f-4003-b35a-62de857fbe80\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " Oct 30 18:23:46.822345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.794346 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-workload-socket\") pod \"31f08f06-6d9f-4003-b35a-62de857fbe80\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " Oct 30 18:23:46.822345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.794426 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-credential-socket\") pod \"31f08f06-6d9f-4003-b35a-62de857fbe80\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " Oct 30 18:23:46.822345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.794540 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-podinfo\") pod \"31f08f06-6d9f-4003-b35a-62de857fbe80\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " Oct 30 18:23:46.822345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.794616 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31f08f06-6d9f-4003-b35a-62de857fbe80-kube-api-access-cj92j\" (UniqueName: \"kubernetes.io/projected/31f08f06-6d9f-4003-b35a-62de857fbe80-kube-api-access-cj92j\") pod \"31f08f06-6d9f-4003-b35a-62de857fbe80\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " Oct 30 18:23:46.822345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.794682 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-workload-certs\") pod \"31f08f06-6d9f-4003-b35a-62de857fbe80\" (UID: \"31f08f06-6d9f-4003-b35a-62de857fbe80\") " Oct 30 18:23:46.822345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.806013 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-credential-socket" pod "31f08f06-6d9f-4003-b35a-62de857fbe80" (UID: "31f08f06-6d9f-4003-b35a-62de857fbe80"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.822345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.806225 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-workload-socket" pod "31f08f06-6d9f-4003-b35a-62de857fbe80" (UID: "31f08f06-6d9f-4003-b35a-62de857fbe80"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.822345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.811018 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-data" pod "31f08f06-6d9f-4003-b35a-62de857fbe80" (UID: "31f08f06-6d9f-4003-b35a-62de857fbe80"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.822345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.811338 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31f08f06-6d9f-4003-b35a-62de857fbe80-istiod-ca-cert" pod "31f08f06-6d9f-4003-b35a-62de857fbe80" (UID: "31f08f06-6d9f-4003-b35a-62de857fbe80"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:46.822345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.814328 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-workload-certs" pod "31f08f06-6d9f-4003-b35a-62de857fbe80" (UID: "31f08f06-6d9f-4003-b35a-62de857fbe80"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.822345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.821709 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-envoy" pod "31f08f06-6d9f-4003-b35a-62de857fbe80" (UID: "31f08f06-6d9f-4003-b35a-62de857fbe80"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.823771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.823210 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b090d280-3716-4098-9330-7fd924785b9d-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.823771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.823241 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b090d280-3716-4098-9330-7fd924785b9d-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.823771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.823258 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xr7tv\" (UniqueName: \"kubernetes.io/projected/b090d280-3716-4098-9330-7fd924785b9d-kube-api-access-xr7tv\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.823771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.823273 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c7a6701-721d-448e-af9a-10216921bf48-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.823771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.823376 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f08f06-6d9f-4003-b35a-62de857fbe80-kube-api-access-cj92j" pod "31f08f06-6d9f-4003-b35a-62de857fbe80" (UID: "31f08f06-6d9f-4003-b35a-62de857fbe80"). InnerVolumeSpecName "kube-api-access-cj92j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:46.830399 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.827240 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-token" pod "31f08f06-6d9f-4003-b35a-62de857fbe80" (UID: "31f08f06-6d9f-4003-b35a-62de857fbe80"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:46.830399 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.827325 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-podinfo" pod "31f08f06-6d9f-4003-b35a-62de857fbe80" (UID: "31f08f06-6d9f-4003-b35a-62de857fbe80"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:23:46.857945 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.857263 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:23:46.860993 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.860192 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:23:46.864848 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.864581 5232 generic.go:356] "Generic (PLEG): container finished" podID="0bbd2e9b-4989-46ca-bf9b-afe2ef671726" containerID="660df6d969acdecc8c744f0e7aab5c6755bb3717fede45654cc9261893ebcbf0" exitCode=0 Oct 30 18:23:46.864848 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.864696 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" Oct 30 18:23:46.867187 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.867123 5232 generic.go:356] "Generic (PLEG): container finished" podID="f8c0e348-2b8d-48e2-9ec0-a21a83db391e" containerID="f15ae61695576b22e3376adff368e02d9edb1454d65fd6cc186ba20f5147687d" exitCode=0 Oct 30 18:23:46.867296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.867227 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" Oct 30 18:23:46.869560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.869130 5232 generic.go:356] "Generic (PLEG): container finished" podID="31f08f06-6d9f-4003-b35a-62de857fbe80" containerID="ce86ab799b5b32e654b04d184f606cafb1c99b890e0d9558fe170b221ea81793" exitCode=0 Oct 30 18:23:46.869560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.869251 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" Oct 30 18:23:46.875931 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.875909 5232 generic.go:356] "Generic (PLEG): container finished" podID="94b06967-baa7-4770-9632-f30f27f1dcae" containerID="f0c3ce9cefec72aa003aad71bbf47d3f013da4f47c46676a85da9c7380cc124c" exitCode=0 Oct 30 18:23:46.880619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.878000 5232 generic.go:356] "Generic (PLEG): container finished" podID="c5923f01-3342-409d-8cc6-49e03be3a4ab" containerID="2061a643b81527aa1f3829aba1ef6a731b19d1013c718e825d1f7a482d4c0cbf" exitCode=0 Oct 30 18:23:46.890555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.886007 5232 scope.go:122] "RemoveContainer" containerID="d1d06a9301d39a9aff6833ac853ebe4bf35f0c4cb09b3e4189fc809b80475cdf" Oct 30 18:23:46.890555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.888357 5232 generic.go:356] "Generic (PLEG): container finished" podID="f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" containerID="7e52361b9c056affff55f26709a0ea054c70cd21d17787aa1e947e64eb17c278" exitCode=0 Oct 30 18:23:46.890555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.888471 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" Oct 30 18:23:46.895354 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.894299 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:23:46.896116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.895691 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" Oct 30 18:23:46.899927 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.896392 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.910395 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" event={"ID":"0bbd2e9b-4989-46ca-bf9b-afe2ef671726","Type":"ContainerDied","Data":"660df6d969acdecc8c744f0e7aab5c6755bb3717fede45654cc9261893ebcbf0"} Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.910450 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6" event={"ID":"0bbd2e9b-4989-46ca-bf9b-afe2ef671726","Type":"ContainerDied","Data":"23a2c4e81093764cc3785180178c87a3cc9c883ad37ac8358fe4b1091d7aef1a"} Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.910474 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" event={"ID":"f8c0e348-2b8d-48e2-9ec0-a21a83db391e","Type":"ContainerDied","Data":"f15ae61695576b22e3376adff368e02d9edb1454d65fd6cc186ba20f5147687d"} Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.910500 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974" event={"ID":"f8c0e348-2b8d-48e2-9ec0-a21a83db391e","Type":"ContainerDied","Data":"32e89529d4fea30aabe02d482886c5910273e88d428c15413b6bf412eeb66942"} Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.910551 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" event={"ID":"31f08f06-6d9f-4003-b35a-62de857fbe80","Type":"ContainerDied","Data":"ce86ab799b5b32e654b04d184f606cafb1c99b890e0d9558fe170b221ea81793"} Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.910579 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247" event={"ID":"31f08f06-6d9f-4003-b35a-62de857fbe80","Type":"ContainerDied","Data":"9c6a997df3b8e911737fe1ed8c4cd77ddd16576e8a2d044f49d52ebb3dab9d89"} Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.910600 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" event={"ID":"94b06967-baa7-4770-9632-f30f27f1dcae","Type":"ContainerDied","Data":"f0c3ce9cefec72aa003aad71bbf47d3f013da4f47c46676a85da9c7380cc124c"} Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.910627 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" event={"ID":"c5923f01-3342-409d-8cc6-49e03be3a4ab","Type":"ContainerDied","Data":"2061a643b81527aa1f3829aba1ef6a731b19d1013c718e825d1f7a482d4c0cbf"} Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.910652 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" event={"ID":"c5923f01-3342-409d-8cc6-49e03be3a4ab","Type":"ContainerDied","Data":"7ac81f87e7d7fa0d084d4babd0ee537328ee36c8932ad305ab18252dcfe078fe"} Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.910675 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" event={"ID":"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0","Type":"ContainerDied","Data":"7e52361b9c056affff55f26709a0ea054c70cd21d17787aa1e947e64eb17c278"} Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.910699 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5" event={"ID":"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0","Type":"ContainerDied","Data":"46ea0c3b3670f4824c0d294e6dcc42d1422b86f6a1718bbc4c09050997c78ef1"} Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.910720 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" event={"ID":"17ccda47-32b1-42f6-9030-0c040fc73883","Type":"ContainerDied","Data":"3ce841e0fa18de102057ae00c5008fbc14828978a439b2255418b864177fff26"} Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.910743 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2" event={"ID":"6c7a6701-721d-448e-af9a-10216921bf48","Type":"ContainerDied","Data":"ebfe8f0f900a962bd67566879a141eb1a901c596afa87da0b72d4c79f97cb871"} Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.910768 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw" event={"ID":"b090d280-3716-4098-9330-7fd924785b9d","Type":"ContainerDied","Data":"1fef86f30660f7a268750342ef36ffc2cb0740ca4fef1f595e5758de3174d617"} Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.910805 5232 scope.go:122] "RemoveContainer" containerID="660df6d969acdecc8c744f0e7aab5c6755bb3717fede45654cc9261893ebcbf0" Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.910999 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk" Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.912561 5232 generic.go:356] "Generic (PLEG): container finished" podID="897d6674-2313-4b21-8a84-d84d56d1c22e" containerID="88b8a3bed368fe3fc9a323daf6ab1b1babcc75f691270e1c60b49f2c8155e453" exitCode=0 Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.913046 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.913194 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" event={"ID":"897d6674-2313-4b21-8a84-d84d56d1c22e","Type":"ContainerDied","Data":"88b8a3bed368fe3fc9a323daf6ab1b1babcc75f691270e1c60b49f2c8155e453"} Oct 30 18:23:46.916355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.913253 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6" event={"ID":"897d6674-2313-4b21-8a84-d84d56d1c22e","Type":"ContainerDied","Data":"3654064f6eedbf5018af6935cd037adeb53be0650830145fba4857ac407e86a3"} Oct 30 18:23:46.920006 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.919951 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.923988 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-token\" (UniqueName: \"kubernetes.io/projected/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-token\") pod \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.924060 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-workload-socket\") pod \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.924140 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-workload-certs\") pod \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.924188 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-credential-socket\") pod \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.924236 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-podinfo\") pod \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.924312 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istiod-ca-cert\") pod \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.924347 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-data\") pod \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.924394 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-envoy\") pod \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.924483 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-workload-certs\") pod \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.924530 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-workload-socket" pod "f8c0e348-2b8d-48e2-9ec0-a21a83db391e" (UID: "f8c0e348-2b8d-48e2-9ec0-a21a83db391e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.924568 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-token\" (UniqueName: \"kubernetes.io/projected/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-token\") pod \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.924619 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-envoy\") pod \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.924649 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-workload-socket\") pod \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.924739 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istiod-ca-cert\") pod \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.924783 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-podinfo\") pod \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.924845 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-kube-api-access-qhjxm\" (UniqueName: \"kubernetes.io/projected/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-kube-api-access-qhjxm\") pod \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\" (UID: \"f8c0e348-2b8d-48e2-9ec0-a21a83db391e\") " Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.924881 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-credential-socket\") pod \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.925002 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-kube-api-access-tr8sh\" (UniqueName: \"kubernetes.io/projected/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-kube-api-access-tr8sh\") pod \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.925071 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-data\") pod \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\" (UID: \"0bbd2e9b-4989-46ca-bf9b-afe2ef671726\") " Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.925986 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/31f08f06-6d9f-4003-b35a-62de857fbe80-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.926009 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.926023 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.926035 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.926047 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.926060 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.926074 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.926087 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/31f08f06-6d9f-4003-b35a-62de857fbe80-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.926099 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cj92j\" (UniqueName: \"kubernetes.io/projected/31f08f06-6d9f-4003-b35a-62de857fbe80-kube-api-access-cj92j\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.926111 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/31f08f06-6d9f-4003-b35a-62de857fbe80-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.926351 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-data" pod "0bbd2e9b-4989-46ca-bf9b-afe2ef671726" (UID: "0bbd2e9b-4989-46ca-bf9b-afe2ef671726"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.926580 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-workload-certs" pod "f8c0e348-2b8d-48e2-9ec0-a21a83db391e" (UID: "f8c0e348-2b8d-48e2-9ec0-a21a83db391e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.926562 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-workload-certs" pod "0bbd2e9b-4989-46ca-bf9b-afe2ef671726" (UID: "0bbd2e9b-4989-46ca-bf9b-afe2ef671726"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.926954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.926934 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-credential-socket" pod "f8c0e348-2b8d-48e2-9ec0-a21a83db391e" (UID: "f8c0e348-2b8d-48e2-9ec0-a21a83db391e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.931770 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.927552 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-data" pod "f8c0e348-2b8d-48e2-9ec0-a21a83db391e" (UID: "f8c0e348-2b8d-48e2-9ec0-a21a83db391e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.931770 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.927845 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istiod-ca-cert" pod "f8c0e348-2b8d-48e2-9ec0-a21a83db391e" (UID: "f8c0e348-2b8d-48e2-9ec0-a21a83db391e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:46.931770 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.930647 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-workload-socket" pod "0bbd2e9b-4989-46ca-bf9b-afe2ef671726" (UID: "0bbd2e9b-4989-46ca-bf9b-afe2ef671726"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.933326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.933303 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-credential-socket" pod "0bbd2e9b-4989-46ca-bf9b-afe2ef671726" (UID: "0bbd2e9b-4989-46ca-bf9b-afe2ef671726"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.934898 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.934872 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istiod-ca-cert" pod "0bbd2e9b-4989-46ca-bf9b-afe2ef671726" (UID: "0bbd2e9b-4989-46ca-bf9b-afe2ef671726"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:46.941918 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.941897 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-envoy" pod "f8c0e348-2b8d-48e2-9ec0-a21a83db391e" (UID: "f8c0e348-2b8d-48e2-9ec0-a21a83db391e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.943944 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.943871 5232 scope.go:122] "RemoveContainer" containerID="660df6d969acdecc8c744f0e7aab5c6755bb3717fede45654cc9261893ebcbf0" Oct 30 18:23:46.944259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.944022 5232 scope.go:122] "RemoveContainer" containerID="ce86ab799b5b32e654b04d184f606cafb1c99b890e0d9558fe170b221ea81793" Oct 30 18:23:46.944259 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:23:46.944188 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"660df6d969acdecc8c744f0e7aab5c6755bb3717fede45654cc9261893ebcbf0\": container with ID starting with 660df6d969acdecc8c744f0e7aab5c6755bb3717fede45654cc9261893ebcbf0 not found: ID does not exist" containerID="660df6d969acdecc8c744f0e7aab5c6755bb3717fede45654cc9261893ebcbf0" Oct 30 18:23:46.944259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.944210 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"660df6d969acdecc8c744f0e7aab5c6755bb3717fede45654cc9261893ebcbf0"} err="failed to get container status \"660df6d969acdecc8c744f0e7aab5c6755bb3717fede45654cc9261893ebcbf0\": rpc error: code = NotFound desc = could not find container \"660df6d969acdecc8c744f0e7aab5c6755bb3717fede45654cc9261893ebcbf0\": container with ID starting with 660df6d969acdecc8c744f0e7aab5c6755bb3717fede45654cc9261893ebcbf0 not found: ID does not exist" Oct 30 18:23:46.944259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.944226 5232 scope.go:122] "RemoveContainer" containerID="f15ae61695576b22e3376adff368e02d9edb1454d65fd6cc186ba20f5147687d" Oct 30 18:23:46.952031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.952005 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-podinfo" pod "f8c0e348-2b8d-48e2-9ec0-a21a83db391e" (UID: "f8c0e348-2b8d-48e2-9ec0-a21a83db391e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:23:46.952658 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.952619 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-envoy" pod "0bbd2e9b-4989-46ca-bf9b-afe2ef671726" (UID: "0bbd2e9b-4989-46ca-bf9b-afe2ef671726"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:46.953205 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.953173 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-kube-api-access-qhjxm" pod "f8c0e348-2b8d-48e2-9ec0-a21a83db391e" (UID: "f8c0e348-2b8d-48e2-9ec0-a21a83db391e"). InnerVolumeSpecName "kube-api-access-qhjxm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:46.963225 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.963197 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-token" pod "0bbd2e9b-4989-46ca-bf9b-afe2ef671726" (UID: "0bbd2e9b-4989-46ca-bf9b-afe2ef671726"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:46.963638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.963614 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-kube-api-access-tr8sh" pod "0bbd2e9b-4989-46ca-bf9b-afe2ef671726" (UID: "0bbd2e9b-4989-46ca-bf9b-afe2ef671726"). InnerVolumeSpecName "kube-api-access-tr8sh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:46.963892 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.963874 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-podinfo" pod "0bbd2e9b-4989-46ca-bf9b-afe2ef671726" (UID: "0bbd2e9b-4989-46ca-bf9b-afe2ef671726"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:23:46.967147 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.966658 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-token" pod "f8c0e348-2b8d-48e2-9ec0-a21a83db391e" (UID: "f8c0e348-2b8d-48e2-9ec0-a21a83db391e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:46.970661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.970638 5232 scope.go:122] "RemoveContainer" containerID="c644ecb172cd9640e48f6b6b1a0f8c1dc380318e906d876cce36c4fecfb5055b" Oct 30 18:23:46.971090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.971070 5232 scope.go:122] "RemoveContainer" containerID="f0c3ce9cefec72aa003aad71bbf47d3f013da4f47c46676a85da9c7380cc124c" Oct 30 18:23:46.986064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.986036 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:46.998831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:46.998810 5232 scope.go:122] "RemoveContainer" containerID="f15ae61695576b22e3376adff368e02d9edb1454d65fd6cc186ba20f5147687d" Oct 30 18:23:47.000180 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.000103 5232 scope.go:122] "RemoveContainer" containerID="47511684f56117e29a19dc627dd6f3a0534e97b882873c5883f784e0b5045392" Oct 30 18:23:47.000180 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:23:47.000145 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f15ae61695576b22e3376adff368e02d9edb1454d65fd6cc186ba20f5147687d\": container with ID starting with f15ae61695576b22e3376adff368e02d9edb1454d65fd6cc186ba20f5147687d not found: ID does not exist" containerID="f15ae61695576b22e3376adff368e02d9edb1454d65fd6cc186ba20f5147687d" Oct 30 18:23:47.000422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.000188 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f15ae61695576b22e3376adff368e02d9edb1454d65fd6cc186ba20f5147687d"} err="failed to get container status \"f15ae61695576b22e3376adff368e02d9edb1454d65fd6cc186ba20f5147687d\": rpc error: code = NotFound desc = could not find container \"f15ae61695576b22e3376adff368e02d9edb1454d65fd6cc186ba20f5147687d\": container with ID starting with f15ae61695576b22e3376adff368e02d9edb1454d65fd6cc186ba20f5147687d not found: ID does not exist" Oct 30 18:23:47.000422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.000211 5232 scope.go:122] "RemoveContainer" containerID="c644ecb172cd9640e48f6b6b1a0f8c1dc380318e906d876cce36c4fecfb5055b" Oct 30 18:23:47.000770 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:23:47.000626 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c644ecb172cd9640e48f6b6b1a0f8c1dc380318e906d876cce36c4fecfb5055b\": container with ID starting with c644ecb172cd9640e48f6b6b1a0f8c1dc380318e906d876cce36c4fecfb5055b not found: ID does not exist" containerID="c644ecb172cd9640e48f6b6b1a0f8c1dc380318e906d876cce36c4fecfb5055b" Oct 30 18:23:47.000770 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.000666 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c644ecb172cd9640e48f6b6b1a0f8c1dc380318e906d876cce36c4fecfb5055b"} err="failed to get container status \"c644ecb172cd9640e48f6b6b1a0f8c1dc380318e906d876cce36c4fecfb5055b\": rpc error: code = NotFound desc = could not find container \"c644ecb172cd9640e48f6b6b1a0f8c1dc380318e906d876cce36c4fecfb5055b\": container with ID starting with c644ecb172cd9640e48f6b6b1a0f8c1dc380318e906d876cce36c4fecfb5055b not found: ID does not exist" Oct 30 18:23:47.000770 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.000689 5232 scope.go:122] "RemoveContainer" containerID="ce86ab799b5b32e654b04d184f606cafb1c99b890e0d9558fe170b221ea81793" Oct 30 18:23:47.002117 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:23:47.001997 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ce86ab799b5b32e654b04d184f606cafb1c99b890e0d9558fe170b221ea81793\": container with ID starting with ce86ab799b5b32e654b04d184f606cafb1c99b890e0d9558fe170b221ea81793 not found: ID does not exist" containerID="ce86ab799b5b32e654b04d184f606cafb1c99b890e0d9558fe170b221ea81793" Oct 30 18:23:47.002117 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.002030 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce86ab799b5b32e654b04d184f606cafb1c99b890e0d9558fe170b221ea81793"} err="failed to get container status \"ce86ab799b5b32e654b04d184f606cafb1c99b890e0d9558fe170b221ea81793\": rpc error: code = NotFound desc = could not find container \"ce86ab799b5b32e654b04d184f606cafb1c99b890e0d9558fe170b221ea81793\": container with ID starting with ce86ab799b5b32e654b04d184f606cafb1c99b890e0d9558fe170b221ea81793 not found: ID does not exist" Oct 30 18:23:47.002117 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.002047 5232 scope.go:122] "RemoveContainer" containerID="ce86ab799b5b32e654b04d184f606cafb1c99b890e0d9558fe170b221ea81793" Oct 30 18:23:47.002826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.002801 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ce86ab799b5b32e654b04d184f606cafb1c99b890e0d9558fe170b221ea81793"} err="failed to get container status \"ce86ab799b5b32e654b04d184f606cafb1c99b890e0d9558fe170b221ea81793\": rpc error: code = NotFound desc = could not find container \"ce86ab799b5b32e654b04d184f606cafb1c99b890e0d9558fe170b221ea81793\": container with ID starting with ce86ab799b5b32e654b04d184f606cafb1c99b890e0d9558fe170b221ea81793 not found: ID does not exist" Oct 30 18:23:47.002917 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.002827 5232 scope.go:122] "RemoveContainer" containerID="2061a643b81527aa1f3829aba1ef6a731b19d1013c718e825d1f7a482d4c0cbf" Oct 30 18:23:47.024622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.024592 5232 scope.go:122] "RemoveContainer" containerID="26cc352eaf761b48087a2b1c5a0c6441ed7a7fc806dd6690436f824266523ca9" Oct 30 18:23:47.025074 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.025058 5232 scope.go:122] "RemoveContainer" containerID="2061a643b81527aa1f3829aba1ef6a731b19d1013c718e825d1f7a482d4c0cbf" Oct 30 18:23:47.025456 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:23:47.025418 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2061a643b81527aa1f3829aba1ef6a731b19d1013c718e825d1f7a482d4c0cbf\": container with ID starting with 2061a643b81527aa1f3829aba1ef6a731b19d1013c718e825d1f7a482d4c0cbf not found: ID does not exist" containerID="2061a643b81527aa1f3829aba1ef6a731b19d1013c718e825d1f7a482d4c0cbf" Oct 30 18:23:47.025569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.025468 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2061a643b81527aa1f3829aba1ef6a731b19d1013c718e825d1f7a482d4c0cbf"} err="failed to get container status \"2061a643b81527aa1f3829aba1ef6a731b19d1013c718e825d1f7a482d4c0cbf\": rpc error: code = NotFound desc = could not find container \"2061a643b81527aa1f3829aba1ef6a731b19d1013c718e825d1f7a482d4c0cbf\": container with ID starting with 2061a643b81527aa1f3829aba1ef6a731b19d1013c718e825d1f7a482d4c0cbf not found: ID does not exist" Oct 30 18:23:47.025569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.025489 5232 scope.go:122] "RemoveContainer" containerID="7e52361b9c056affff55f26709a0ea054c70cd21d17787aa1e947e64eb17c278" Oct 30 18:23:47.026872 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.026847 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/94b06967-baa7-4770-9632-f30f27f1dcae-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b06967-baa7-4770-9632-f30f27f1dcae-istiod-ca-cert\") pod \"94b06967-baa7-4770-9632-f30f27f1dcae\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " Oct 30 18:23:47.026974 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.026904 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b06967-baa7-4770-9632-f30f27f1dcae-kube-api-access-czr2r\" (UniqueName: \"kubernetes.io/projected/94b06967-baa7-4770-9632-f30f27f1dcae-kube-api-access-czr2r\") pod \"94b06967-baa7-4770-9632-f30f27f1dcae\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " Oct 30 18:23:47.026974 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.026952 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-credential-socket\") pod \"897d6674-2313-4b21-8a84-d84d56d1c22e\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " Oct 30 18:23:47.027116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.026999 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-workload-socket\") pod \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " Oct 30 18:23:47.027116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027038 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-data\") pod \"c5923f01-3342-409d-8cc6-49e03be3a4ab\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " Oct 30 18:23:47.027226 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027115 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-workload-socket\") pod \"94b06967-baa7-4770-9632-f30f27f1dcae\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " Oct 30 18:23:47.027226 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027161 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-istio-envoy\") pod \"94b06967-baa7-4770-9632-f30f27f1dcae\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " Oct 30 18:23:47.027226 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027212 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/897d6674-2313-4b21-8a84-d84d56d1c22e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/897d6674-2313-4b21-8a84-d84d56d1c22e-istiod-ca-cert\") pod \"897d6674-2313-4b21-8a84-d84d56d1c22e\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " Oct 30 18:23:47.027388 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027242 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-data\") pod \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " Oct 30 18:23:47.027388 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027281 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-credential-socket" pod "897d6674-2313-4b21-8a84-d84d56d1c22e" (UID: "897d6674-2313-4b21-8a84-d84d56d1c22e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.027388 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027298 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-workload-socket" pod "f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" (UID: "f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.027388 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027288 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-kube-api-access-9qfbm\" (UniqueName: \"kubernetes.io/projected/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-kube-api-access-9qfbm\") pod \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " Oct 30 18:23:47.027388 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027308 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/94b06967-baa7-4770-9632-f30f27f1dcae-istiod-ca-cert" pod "94b06967-baa7-4770-9632-f30f27f1dcae" (UID: "94b06967-baa7-4770-9632-f30f27f1dcae"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:47.027388 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027357 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-workload-certs\") pod \"c5923f01-3342-409d-8cc6-49e03be3a4ab\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " Oct 30 18:23:47.027774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027401 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5923f01-3342-409d-8cc6-49e03be3a4ab-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5923f01-3342-409d-8cc6-49e03be3a4ab-istiod-ca-cert\") pod \"c5923f01-3342-409d-8cc6-49e03be3a4ab\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " Oct 30 18:23:47.027774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027448 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-istio-data\") pod \"94b06967-baa7-4770-9632-f30f27f1dcae\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " Oct 30 18:23:47.027774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027523 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-podinfo\") pod \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " Oct 30 18:23:47.027774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027550 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-workload-certs\") pod \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " Oct 30 18:23:47.027774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027605 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/94b06967-baa7-4770-9632-f30f27f1dcae-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b06967-baa7-4770-9632-f30f27f1dcae-istio-podinfo\") pod \"94b06967-baa7-4770-9632-f30f27f1dcae\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " Oct 30 18:23:47.027774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027647 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-workload-certs\") pod \"94b06967-baa7-4770-9632-f30f27f1dcae\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " Oct 30 18:23:47.027774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027699 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/94b06967-baa7-4770-9632-f30f27f1dcae-istio-token\" (UniqueName: \"kubernetes.io/projected/94b06967-baa7-4770-9632-f30f27f1dcae-istio-token\") pod \"94b06967-baa7-4770-9632-f30f27f1dcae\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " Oct 30 18:23:47.027774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027738 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/897d6674-2313-4b21-8a84-d84d56d1c22e-istiod-ca-cert" pod "897d6674-2313-4b21-8a84-d84d56d1c22e" (UID: "897d6674-2313-4b21-8a84-d84d56d1c22e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:47.027774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027743 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-data\") pod \"897d6674-2313-4b21-8a84-d84d56d1c22e\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " Oct 30 18:23:47.028250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027811 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5923f01-3342-409d-8cc6-49e03be3a4ab-kube-api-access-2hzwp\" (UniqueName: \"kubernetes.io/projected/c5923f01-3342-409d-8cc6-49e03be3a4ab-kube-api-access-2hzwp\") pod \"c5923f01-3342-409d-8cc6-49e03be3a4ab\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " Oct 30 18:23:47.028250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027849 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-envoy\") pod \"c5923f01-3342-409d-8cc6-49e03be3a4ab\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " Oct 30 18:23:47.028250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027885 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-podinfo\") pod \"897d6674-2313-4b21-8a84-d84d56d1c22e\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " Oct 30 18:23:47.028250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027916 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-workload-socket\") pod \"c5923f01-3342-409d-8cc6-49e03be3a4ab\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " Oct 30 18:23:47.028250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.028013 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-envoy\") pod \"897d6674-2313-4b21-8a84-d84d56d1c22e\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " Oct 30 18:23:47.028250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.028057 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-credential-socket\") pod \"c5923f01-3342-409d-8cc6-49e03be3a4ab\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " Oct 30 18:23:47.028250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.028119 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-token\" (UniqueName: \"kubernetes.io/projected/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-token\") pod \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " Oct 30 18:23:47.028250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.028198 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-podinfo\") pod \"c5923f01-3342-409d-8cc6-49e03be3a4ab\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " Oct 30 18:23:47.028743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.028266 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-token\" (UniqueName: \"kubernetes.io/projected/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-token\") pod \"897d6674-2313-4b21-8a84-d84d56d1c22e\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " Oct 30 18:23:47.028743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.028311 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-envoy\") pod \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " Oct 30 18:23:47.028743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.028351 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-credential-socket\") pod \"94b06967-baa7-4770-9632-f30f27f1dcae\" (UID: \"94b06967-baa7-4770-9632-f30f27f1dcae\") " Oct 30 18:23:47.028743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.028404 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/897d6674-2313-4b21-8a84-d84d56d1c22e-kube-api-access-rqs8v\" (UniqueName: \"kubernetes.io/projected/897d6674-2313-4b21-8a84-d84d56d1c22e-kube-api-access-rqs8v\") pod \"897d6674-2313-4b21-8a84-d84d56d1c22e\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " Oct 30 18:23:47.028743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.028469 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-credential-socket\") pod \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " Oct 30 18:23:47.028743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.028496 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-token\" (UniqueName: \"kubernetes.io/projected/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-token\") pod \"c5923f01-3342-409d-8cc6-49e03be3a4ab\" (UID: \"c5923f01-3342-409d-8cc6-49e03be3a4ab\") " Oct 30 18:23:47.028743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.028665 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istiod-ca-cert\") pod \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\" (UID: \"f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0\") " Oct 30 18:23:47.028743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.028727 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-workload-socket\") pod \"897d6674-2313-4b21-8a84-d84d56d1c22e\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " Oct 30 18:23:47.029170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.028758 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-workload-certs\") pod \"897d6674-2313-4b21-8a84-d84d56d1c22e\" (UID: \"897d6674-2313-4b21-8a84-d84d56d1c22e\") " Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030636 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030661 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030696 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030712 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030724 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030738 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030773 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhjxm\" (UniqueName: \"kubernetes.io/projected/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-kube-api-access-qhjxm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030789 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030803 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tr8sh\" (UniqueName: \"kubernetes.io/projected/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-kube-api-access-tr8sh\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030816 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030848 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/94b06967-baa7-4770-9632-f30f27f1dcae-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030863 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030880 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030894 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030907 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030945 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030959 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0bbd2e9b-4989-46ca-bf9b-afe2ef671726-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030973 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/897d6674-2313-4b21-8a84-d84d56d1c22e-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030986 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.031023 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.033660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.031036 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f8c0e348-2b8d-48e2-9ec0-a21a83db391e-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.040583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.027735 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-data" pod "f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" (UID: "f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.040583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.028083 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5923f01-3342-409d-8cc6-49e03be3a4ab-istiod-ca-cert" pod "c5923f01-3342-409d-8cc6-49e03be3a4ab" (UID: "c5923f01-3342-409d-8cc6-49e03be3a4ab"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:47.040583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.028147 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-workload-certs" pod "94b06967-baa7-4770-9632-f30f27f1dcae" (UID: "94b06967-baa7-4770-9632-f30f27f1dcae"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.040583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.028285 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-workload-certs" pod "f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" (UID: "f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.040583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.028528 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-istio-data" pod "94b06967-baa7-4770-9632-f30f27f1dcae" (UID: "94b06967-baa7-4770-9632-f30f27f1dcae"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.040583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.029084 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-credential-socket" pod "f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" (UID: "f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.040583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.029869 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-workload-certs" pod "c5923f01-3342-409d-8cc6-49e03be3a4ab" (UID: "c5923f01-3342-409d-8cc6-49e03be3a4ab"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.040583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.030979 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-podinfo" pod "f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" (UID: "f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:23:47.040583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.031903 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-data" pod "c5923f01-3342-409d-8cc6-49e03be3a4ab" (UID: "c5923f01-3342-409d-8cc6-49e03be3a4ab"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.040583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.033711 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-workload-socket" pod "94b06967-baa7-4770-9632-f30f27f1dcae" (UID: "94b06967-baa7-4770-9632-f30f27f1dcae"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.040583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.034911 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-workload-certs" pod "897d6674-2313-4b21-8a84-d84d56d1c22e" (UID: "897d6674-2313-4b21-8a84-d84d56d1c22e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.040583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.035936 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-workload-socket" pod "897d6674-2313-4b21-8a84-d84d56d1c22e" (UID: "897d6674-2313-4b21-8a84-d84d56d1c22e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.040583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.036081 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-data" pod "897d6674-2313-4b21-8a84-d84d56d1c22e" (UID: "897d6674-2313-4b21-8a84-d84d56d1c22e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.040583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.037142 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istiod-ca-cert" pod "f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" (UID: "f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:23:47.040583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.037191 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-credential-socket" pod "94b06967-baa7-4770-9632-f30f27f1dcae" (UID: "94b06967-baa7-4770-9632-f30f27f1dcae"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.040583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.040004 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-workload-socket" pod "c5923f01-3342-409d-8cc6-49e03be3a4ab" (UID: "c5923f01-3342-409d-8cc6-49e03be3a4ab"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.041331 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.041311 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-credential-socket" pod "c5923f01-3342-409d-8cc6-49e03be3a4ab" (UID: "c5923f01-3342-409d-8cc6-49e03be3a4ab"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.047025 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.046989 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5923f01-3342-409d-8cc6-49e03be3a4ab-kube-api-access-2hzwp" pod "c5923f01-3342-409d-8cc6-49e03be3a4ab" (UID: "c5923f01-3342-409d-8cc6-49e03be3a4ab"). InnerVolumeSpecName "kube-api-access-2hzwp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:47.047494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.047435 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/94b06967-baa7-4770-9632-f30f27f1dcae-istio-podinfo" pod "94b06967-baa7-4770-9632-f30f27f1dcae" (UID: "94b06967-baa7-4770-9632-f30f27f1dcae"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:23:47.047610 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.047567 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-envoy" pod "897d6674-2313-4b21-8a84-d84d56d1c22e" (UID: "897d6674-2313-4b21-8a84-d84d56d1c22e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.047853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.047794 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-istio-envoy" pod "94b06967-baa7-4770-9632-f30f27f1dcae" (UID: "94b06967-baa7-4770-9632-f30f27f1dcae"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.047853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.047823 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-token" pod "f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" (UID: "f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:47.047990 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.047947 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-envoy" pod "c5923f01-3342-409d-8cc6-49e03be3a4ab" (UID: "c5923f01-3342-409d-8cc6-49e03be3a4ab"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.048137 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.048118 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-kube-api-access-9qfbm" pod "f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" (UID: "f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0"). InnerVolumeSpecName "kube-api-access-9qfbm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:47.050991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.050964 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-podinfo" pod "c5923f01-3342-409d-8cc6-49e03be3a4ab" (UID: "c5923f01-3342-409d-8cc6-49e03be3a4ab"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:23:47.051356 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.051245 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-envoy" pod "f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" (UID: "f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:23:47.051356 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.051329 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-token" pod "897d6674-2313-4b21-8a84-d84d56d1c22e" (UID: "897d6674-2313-4b21-8a84-d84d56d1c22e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:47.051663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.051641 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/897d6674-2313-4b21-8a84-d84d56d1c22e-kube-api-access-rqs8v" pod "897d6674-2313-4b21-8a84-d84d56d1c22e" (UID: "897d6674-2313-4b21-8a84-d84d56d1c22e"). InnerVolumeSpecName "kube-api-access-rqs8v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:47.052000 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.051980 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-token" pod "c5923f01-3342-409d-8cc6-49e03be3a4ab" (UID: "c5923f01-3342-409d-8cc6-49e03be3a4ab"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:47.052907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.052883 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b06967-baa7-4770-9632-f30f27f1dcae-istio-token" pod "94b06967-baa7-4770-9632-f30f27f1dcae" (UID: "94b06967-baa7-4770-9632-f30f27f1dcae"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:47.053271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.053246 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/94b06967-baa7-4770-9632-f30f27f1dcae-kube-api-access-czr2r" pod "94b06967-baa7-4770-9632-f30f27f1dcae" (UID: "94b06967-baa7-4770-9632-f30f27f1dcae"). InnerVolumeSpecName "kube-api-access-czr2r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:23:47.056095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.056074 5232 scope.go:122] "RemoveContainer" containerID="f7831da1c6a0eaa1c0005c8a86deee45a7e77459e3b9edf3dfff91079a9bab3d" Oct 30 18:23:47.058996 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.058977 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-podinfo" pod "897d6674-2313-4b21-8a84-d84d56d1c22e" (UID: "897d6674-2313-4b21-8a84-d84d56d1c22e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:23:47.060170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.060151 5232 scope.go:122] "RemoveContainer" containerID="7e52361b9c056affff55f26709a0ea054c70cd21d17787aa1e947e64eb17c278" Oct 30 18:23:47.061832 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:23:47.060480 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7e52361b9c056affff55f26709a0ea054c70cd21d17787aa1e947e64eb17c278\": container with ID starting with 7e52361b9c056affff55f26709a0ea054c70cd21d17787aa1e947e64eb17c278 not found: ID does not exist" containerID="7e52361b9c056affff55f26709a0ea054c70cd21d17787aa1e947e64eb17c278" Oct 30 18:23:47.061832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.060572 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7e52361b9c056affff55f26709a0ea054c70cd21d17787aa1e947e64eb17c278"} err="failed to get container status \"7e52361b9c056affff55f26709a0ea054c70cd21d17787aa1e947e64eb17c278\": rpc error: code = NotFound desc = could not find container \"7e52361b9c056affff55f26709a0ea054c70cd21d17787aa1e947e64eb17c278\": container with ID starting with 7e52361b9c056affff55f26709a0ea054c70cd21d17787aa1e947e64eb17c278 not found: ID does not exist" Oct 30 18:23:47.061832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.060597 5232 scope.go:122] "RemoveContainer" containerID="4762ad491b3bde8ce19235625990fe0e46e2ab599ccc9dbc6bb619b096b441d2" Oct 30 18:23:47.076560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.076479 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2"] Oct 30 18:23:47.079422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.079404 5232 scope.go:122] "RemoveContainer" containerID="a4394484a4fc36dc9bdb68c258090b3c628f4ac38b3e80e50f96bb5451abaf4d" Oct 30 18:23:47.080587 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.080500 5232 scope.go:122] "RemoveContainer" containerID="8eb2dc9ae29604190a5d4a178668d778f78580546111260f2e2e59af72d54162" Oct 30 18:23:47.084362 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.084340 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-qnkb2"] Oct 30 18:23:47.096103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.096086 5232 scope.go:122] "RemoveContainer" containerID="4f27afc9fb40171ec21b0c5657cf3d798240d87e1fcf0621a1b6ce9556682cb5" Oct 30 18:23:47.096560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.096546 5232 scope.go:122] "RemoveContainer" containerID="0738da3cf01dbc44ff0bd79d3bd5904295e1190f6f4cf1d1e60fd38045bc5641" Oct 30 18:23:47.113983 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.113958 5232 scope.go:122] "RemoveContainer" containerID="88b8a3bed368fe3fc9a323daf6ab1b1babcc75f691270e1c60b49f2c8155e453" Oct 30 18:23:47.114362 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.114341 5232 scope.go:122] "RemoveContainer" containerID="88b8a3bed368fe3fc9a323daf6ab1b1babcc75f691270e1c60b49f2c8155e453" Oct 30 18:23:47.131368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.131344 5232 scope.go:122] "RemoveContainer" containerID="d9604a65a31793dc4e37835ca1b7e6e4ea7cce8c41fc14d999414cff0f2f7cd4" Oct 30 18:23:47.132888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.132860 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czr2r\" (UniqueName: \"kubernetes.io/projected/94b06967-baa7-4770-9632-f30f27f1dcae-kube-api-access-czr2r\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.133014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.132892 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.133014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.132907 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.133014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.132923 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.133014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.132936 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.133014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.132948 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qfbm\" (UniqueName: \"kubernetes.io/projected/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-kube-api-access-9qfbm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.133014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.132962 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.133014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.132976 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5923f01-3342-409d-8cc6-49e03be3a4ab-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.133014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.132988 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.133014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133000 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.133014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133012 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133024 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/94b06967-baa7-4770-9632-f30f27f1dcae-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133038 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133051 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/94b06967-baa7-4770-9632-f30f27f1dcae-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133063 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133076 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2hzwp\" (UniqueName: \"kubernetes.io/projected/c5923f01-3342-409d-8cc6-49e03be3a4ab-kube-api-access-2hzwp\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133088 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133101 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133113 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133126 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133138 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5923f01-3342-409d-8cc6-49e03be3a4ab-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133150 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133163 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133175 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/897d6674-2313-4b21-8a84-d84d56d1c22e-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133189 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133202 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/94b06967-baa7-4770-9632-f30f27f1dcae-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133214 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rqs8v\" (UniqueName: \"kubernetes.io/projected/897d6674-2313-4b21-8a84-d84d56d1c22e-kube-api-access-rqs8v\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133228 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133241 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5923f01-3342-409d-8cc6-49e03be3a4ab-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133254 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133266 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.134524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.133279 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/897d6674-2313-4b21-8a84-d84d56d1c22e-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:23:47.135677 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:23:47.134527 5232 log.go:32] "RemoveContainer from runtime service failed" err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6_openshift-ingress_897d6674-2313-4b21-8a84-d84d56d1c22e_0 in pod sandbox 3654064f6eedbf5018af6935cd037adeb53be0650830145fba4857ac407e86a3 from index: no such id: '88b8a3bed368fe3fc9a323daf6ab1b1babcc75f691270e1c60b49f2c8155e453'" containerID="88b8a3bed368fe3fc9a323daf6ab1b1babcc75f691270e1c60b49f2c8155e453" Oct 30 18:23:47.135677 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.134569 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b8a3bed368fe3fc9a323daf6ab1b1babcc75f691270e1c60b49f2c8155e453"} err="rpc error: code = Unknown desc = failed to delete container k8s_istio-proxy_e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6_openshift-ingress_897d6674-2313-4b21-8a84-d84d56d1c22e_0 in pod sandbox 3654064f6eedbf5018af6935cd037adeb53be0650830145fba4857ac407e86a3 from index: no such id: '88b8a3bed368fe3fc9a323daf6ab1b1babcc75f691270e1c60b49f2c8155e453'" Oct 30 18:23:47.135677 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.134611 5232 scope.go:122] "RemoveContainer" containerID="88b8a3bed368fe3fc9a323daf6ab1b1babcc75f691270e1c60b49f2c8155e453" Oct 30 18:23:47.135677 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:23:47.135152 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88b8a3bed368fe3fc9a323daf6ab1b1babcc75f691270e1c60b49f2c8155e453\": container with ID starting with 88b8a3bed368fe3fc9a323daf6ab1b1babcc75f691270e1c60b49f2c8155e453 not found: ID does not exist" containerID="88b8a3bed368fe3fc9a323daf6ab1b1babcc75f691270e1c60b49f2c8155e453" Oct 30 18:23:47.135677 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.135185 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88b8a3bed368fe3fc9a323daf6ab1b1babcc75f691270e1c60b49f2c8155e453"} err="failed to get container status \"88b8a3bed368fe3fc9a323daf6ab1b1babcc75f691270e1c60b49f2c8155e453\": rpc error: code = NotFound desc = could not find container \"88b8a3bed368fe3fc9a323daf6ab1b1babcc75f691270e1c60b49f2c8155e453\": container with ID starting with 88b8a3bed368fe3fc9a323daf6ab1b1babcc75f691270e1c60b49f2c8155e453 not found: ID does not exist" Oct 30 18:23:47.147389 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.147367 5232 scope.go:122] "RemoveContainer" containerID="5893347a5d0f62fb0eebec7d6b32e43a72ef938cd9f1e65cd701bf06b65e6a17" Oct 30 18:23:47.162037 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.162013 5232 scope.go:122] "RemoveContainer" containerID="bd71115cf7337925862094d9f89e095ffbaf5bdf742d233b6bdd9b705ec557c1" Oct 30 18:23:47.172242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.172218 5232 scope.go:122] "RemoveContainer" containerID="9362570ddb57dfb8b5791500f00f400608cc41eb03e8e637525efee0be59d980" Oct 30 18:23:47.191734 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.191714 5232 scope.go:122] "RemoveContainer" containerID="8eb2dc9ae29604190a5d4a178668d778f78580546111260f2e2e59af72d54162" Oct 30 18:23:47.192154 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:23:47.192125 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8eb2dc9ae29604190a5d4a178668d778f78580546111260f2e2e59af72d54162\": container with ID starting with 8eb2dc9ae29604190a5d4a178668d778f78580546111260f2e2e59af72d54162 not found: ID does not exist" containerID="8eb2dc9ae29604190a5d4a178668d778f78580546111260f2e2e59af72d54162" Oct 30 18:23:47.192270 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:23:47.192162 5232 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"8eb2dc9ae29604190a5d4a178668d778f78580546111260f2e2e59af72d54162\": rpc error: code = NotFound desc = could not find container \"8eb2dc9ae29604190a5d4a178668d778f78580546111260f2e2e59af72d54162\": container with ID starting with 8eb2dc9ae29604190a5d4a178668d778f78580546111260f2e2e59af72d54162 not found: ID does not exist" containerID="8eb2dc9ae29604190a5d4a178668d778f78580546111260f2e2e59af72d54162" Oct 30 18:23:47.192270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.192188 5232 scope.go:122] "RemoveContainer" containerID="c644ecb172cd9640e48f6b6b1a0f8c1dc380318e906d876cce36c4fecfb5055b" Oct 30 18:23:47.192519 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:23:47.192480 5232 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"c644ecb172cd9640e48f6b6b1a0f8c1dc380318e906d876cce36c4fecfb5055b\": rpc error: code = NotFound desc = could not find container \"c644ecb172cd9640e48f6b6b1a0f8c1dc380318e906d876cce36c4fecfb5055b\": container with ID starting with c644ecb172cd9640e48f6b6b1a0f8c1dc380318e906d876cce36c4fecfb5055b not found: ID does not exist" containerID="c644ecb172cd9640e48f6b6b1a0f8c1dc380318e906d876cce36c4fecfb5055b" Oct 30 18:23:47.192590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.192555 5232 scope.go:122] "RemoveContainer" containerID="f15ae61695576b22e3376adff368e02d9edb1454d65fd6cc186ba20f5147687d" Oct 30 18:23:47.192843 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:23:47.192820 5232 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"f15ae61695576b22e3376adff368e02d9edb1454d65fd6cc186ba20f5147687d\": rpc error: code = NotFound desc = could not find container \"f15ae61695576b22e3376adff368e02d9edb1454d65fd6cc186ba20f5147687d\": container with ID starting with f15ae61695576b22e3376adff368e02d9edb1454d65fd6cc186ba20f5147687d not found: ID does not exist" containerID="f15ae61695576b22e3376adff368e02d9edb1454d65fd6cc186ba20f5147687d" Oct 30 18:23:47.192933 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.192849 5232 scope.go:122] "RemoveContainer" containerID="99b55c1c67b8a7e76d76e29d3ffa13334b02da569f35c885d013eba7ea960965" Oct 30 18:23:47.215146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.215118 5232 scope.go:122] "RemoveContainer" containerID="660df6d969acdecc8c744f0e7aab5c6755bb3717fede45654cc9261893ebcbf0" Oct 30 18:23:47.215761 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:23:47.215736 5232 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"660df6d969acdecc8c744f0e7aab5c6755bb3717fede45654cc9261893ebcbf0\": rpc error: code = NotFound desc = could not find container \"660df6d969acdecc8c744f0e7aab5c6755bb3717fede45654cc9261893ebcbf0\": container with ID starting with 660df6d969acdecc8c744f0e7aab5c6755bb3717fede45654cc9261893ebcbf0 not found: ID does not exist" containerID="660df6d969acdecc8c744f0e7aab5c6755bb3717fede45654cc9261893ebcbf0" Oct 30 18:23:47.215883 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.215764 5232 scope.go:122] "RemoveContainer" containerID="eb16d995657784c42efcf9ed7bb2b0b7ba4d4e315302e2685b6d60a95d175948" Oct 30 18:23:47.229541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.229502 5232 scope.go:122] "RemoveContainer" containerID="f7aa6e1a61763531b99e5980d1a7c0d66be0ad9030398bb603a5d222ef1eeaf1" Oct 30 18:23:47.241293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.241266 5232 scope.go:122] "RemoveContainer" containerID="4f27afc9fb40171ec21b0c5657cf3d798240d87e1fcf0621a1b6ce9556682cb5" Oct 30 18:23:47.241645 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:23:47.241618 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4f27afc9fb40171ec21b0c5657cf3d798240d87e1fcf0621a1b6ce9556682cb5\": container with ID starting with 4f27afc9fb40171ec21b0c5657cf3d798240d87e1fcf0621a1b6ce9556682cb5 not found: ID does not exist" containerID="4f27afc9fb40171ec21b0c5657cf3d798240d87e1fcf0621a1b6ce9556682cb5" Oct 30 18:23:47.241721 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:23:47.241655 5232 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"4f27afc9fb40171ec21b0c5657cf3d798240d87e1fcf0621a1b6ce9556682cb5\": rpc error: code = NotFound desc = could not find container \"4f27afc9fb40171ec21b0c5657cf3d798240d87e1fcf0621a1b6ce9556682cb5\": container with ID starting with 4f27afc9fb40171ec21b0c5657cf3d798240d87e1fcf0621a1b6ce9556682cb5 not found: ID does not exist" containerID="4f27afc9fb40171ec21b0c5657cf3d798240d87e1fcf0621a1b6ce9556682cb5" Oct 30 18:23:47.241721 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.241679 5232 scope.go:122] "RemoveContainer" containerID="7e52361b9c056affff55f26709a0ea054c70cd21d17787aa1e947e64eb17c278" Oct 30 18:23:47.241944 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:23:47.241924 5232 kuberuntime_gc.go:151] "Failed to remove container" err="failed to get container status \"7e52361b9c056affff55f26709a0ea054c70cd21d17787aa1e947e64eb17c278\": rpc error: code = NotFound desc = could not find container \"7e52361b9c056affff55f26709a0ea054c70cd21d17787aa1e947e64eb17c278\": container with ID starting with 7e52361b9c056affff55f26709a0ea054c70cd21d17787aa1e947e64eb17c278 not found: ID does not exist" containerID="7e52361b9c056affff55f26709a0ea054c70cd21d17787aa1e947e64eb17c278" Oct 30 18:23:47.272024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.271994 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw"] Oct 30 18:23:47.276750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.276684 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-nzlgw"] Oct 30 18:23:47.523567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.523528 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk"] Oct 30 18:23:47.650848 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.649864 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-ntvlk"] Oct 30 18:23:47.660446 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.660373 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247"] Oct 30 18:23:47.670322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.670283 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-7n247"] Oct 30 18:23:47.700796 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.700758 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6"] Oct 30 18:23:47.707121 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.705416 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-vwhw6"] Oct 30 18:23:47.712734 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.712712 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5"] Oct 30 18:23:47.721028 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.720996 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-tgsq5"] Oct 30 18:23:47.728801 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.728784 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974"] Oct 30 18:23:47.735587 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.735538 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-6ce5eb-openshift-default-5d7f7ff9d9-fs974"] Oct 30 18:23:47.739705 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.739680 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6"] Oct 30 18:23:47.744942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.744919 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-mtnt6"] Oct 30 18:23:47.922481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.922399 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" event={"ID":"94b06967-baa7-4770-9632-f30f27f1dcae","Type":"ContainerDied","Data":"db92124d02160507e9e75851a18e0fb12af7d9a8b01348712a06c6ca07f40ff0"} Oct 30 18:23:47.922726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.922697 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527" Oct 30 18:23:47.923031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.922991 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj" Oct 30 18:23:47.923031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.923018 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:47.923931 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:47.923914 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:23:48.042009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:48.041977 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj"] Oct 30 18:23:48.045901 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:48.045875 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-4dfsj"] Oct 30 18:23:48.109283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:48.109245 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527"] Oct 30 18:23:48.139213 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:48.139153 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-l9527"] Oct 30 18:23:48.808330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:48.808296 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0bbd2e9b-4989-46ca-bf9b-afe2ef671726" path="/var/lib/kubelet/pods/0bbd2e9b-4989-46ca-bf9b-afe2ef671726/volumes" Oct 30 18:23:48.809736 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:48.809714 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17ccda47-32b1-42f6-9030-0c040fc73883" path="/var/lib/kubelet/pods/17ccda47-32b1-42f6-9030-0c040fc73883/volumes" Oct 30 18:23:48.810690 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:48.810671 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31f08f06-6d9f-4003-b35a-62de857fbe80" path="/var/lib/kubelet/pods/31f08f06-6d9f-4003-b35a-62de857fbe80/volumes" Oct 30 18:23:48.811986 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:48.811967 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c7a6701-721d-448e-af9a-10216921bf48" path="/var/lib/kubelet/pods/6c7a6701-721d-448e-af9a-10216921bf48/volumes" Oct 30 18:23:48.812874 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:48.812857 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="897d6674-2313-4b21-8a84-d84d56d1c22e" path="/var/lib/kubelet/pods/897d6674-2313-4b21-8a84-d84d56d1c22e/volumes" Oct 30 18:23:48.813752 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:48.813735 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="94b06967-baa7-4770-9632-f30f27f1dcae" path="/var/lib/kubelet/pods/94b06967-baa7-4770-9632-f30f27f1dcae/volumes" Oct 30 18:23:48.814994 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:48.814975 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b090d280-3716-4098-9330-7fd924785b9d" path="/var/lib/kubelet/pods/b090d280-3716-4098-9330-7fd924785b9d/volumes" Oct 30 18:23:48.815867 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:48.815850 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5923f01-3342-409d-8cc6-49e03be3a4ab" path="/var/lib/kubelet/pods/c5923f01-3342-409d-8cc6-49e03be3a4ab/volumes" Oct 30 18:23:48.816863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:48.816839 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" path="/var/lib/kubelet/pods/f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0/volumes" Oct 30 18:23:48.817678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:48.817656 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8c0e348-2b8d-48e2-9ec0-a21a83db391e" path="/var/lib/kubelet/pods/f8c0e348-2b8d-48e2-9ec0-a21a83db391e/volumes" Oct 30 18:23:50.249196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.249167 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt"] Oct 30 18:23:50.250069 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250051 5232 cpu_manager.go:414] "containerMap: removing container" podUID="17ccda47-32b1-42f6-9030-0c040fc73883" containerName="istio-proxy" Oct 30 18:23:50.250131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250070 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17ccda47-32b1-42f6-9030-0c040fc73883" containerName="istio-proxy" Oct 30 18:23:50.250131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250085 5232 cpu_manager.go:414] "containerMap: removing container" podUID="f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" containerName="istio-proxy" Oct 30 18:23:50.250131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250091 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" containerName="istio-proxy" Oct 30 18:23:50.250131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250099 5232 cpu_manager.go:414] "containerMap: removing container" podUID="31f08f06-6d9f-4003-b35a-62de857fbe80" containerName="istio-proxy" Oct 30 18:23:50.250131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250105 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31f08f06-6d9f-4003-b35a-62de857fbe80" containerName="istio-proxy" Oct 30 18:23:50.250131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250115 5232 cpu_manager.go:414] "containerMap: removing container" podUID="94b06967-baa7-4770-9632-f30f27f1dcae" containerName="istio-proxy" Oct 30 18:23:50.250131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250122 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="94b06967-baa7-4770-9632-f30f27f1dcae" containerName="istio-proxy" Oct 30 18:23:50.250364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250152 5232 cpu_manager.go:414] "containerMap: removing container" podUID="f8c0e348-2b8d-48e2-9ec0-a21a83db391e" containerName="istio-proxy" Oct 30 18:23:50.250364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250157 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8c0e348-2b8d-48e2-9ec0-a21a83db391e" containerName="istio-proxy" Oct 30 18:23:50.250364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250164 5232 cpu_manager.go:414] "containerMap: removing container" podUID="f8c0e348-2b8d-48e2-9ec0-a21a83db391e" containerName="istio-proxy" Oct 30 18:23:50.250364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250168 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8c0e348-2b8d-48e2-9ec0-a21a83db391e" containerName="istio-proxy" Oct 30 18:23:50.250364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250176 5232 cpu_manager.go:414] "containerMap: removing container" podUID="0bbd2e9b-4989-46ca-bf9b-afe2ef671726" containerName="istio-proxy" Oct 30 18:23:50.250364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250180 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0bbd2e9b-4989-46ca-bf9b-afe2ef671726" containerName="istio-proxy" Oct 30 18:23:50.250364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250192 5232 cpu_manager.go:414] "containerMap: removing container" podUID="c5923f01-3342-409d-8cc6-49e03be3a4ab" containerName="istio-proxy" Oct 30 18:23:50.250364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250196 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5923f01-3342-409d-8cc6-49e03be3a4ab" containerName="istio-proxy" Oct 30 18:23:50.250364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250210 5232 cpu_manager.go:414] "containerMap: removing container" podUID="6c7a6701-721d-448e-af9a-10216921bf48" containerName="istio-proxy" Oct 30 18:23:50.250364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250213 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c7a6701-721d-448e-af9a-10216921bf48" containerName="istio-proxy" Oct 30 18:23:50.250364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250221 5232 cpu_manager.go:414] "containerMap: removing container" podUID="897d6674-2313-4b21-8a84-d84d56d1c22e" containerName="istio-proxy" Oct 30 18:23:50.250364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250225 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="897d6674-2313-4b21-8a84-d84d56d1c22e" containerName="istio-proxy" Oct 30 18:23:50.250364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250237 5232 cpu_manager.go:414] "containerMap: removing container" podUID="b090d280-3716-4098-9330-7fd924785b9d" containerName="istio-proxy" Oct 30 18:23:50.250364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250241 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b090d280-3716-4098-9330-7fd924785b9d" containerName="istio-proxy" Oct 30 18:23:50.250364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250359 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="897d6674-2313-4b21-8a84-d84d56d1c22e" containerName="istio-proxy" Oct 30 18:23:50.250364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250370 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="b090d280-3716-4098-9330-7fd924785b9d" containerName="istio-proxy" Oct 30 18:23:50.250844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250380 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="17ccda47-32b1-42f6-9030-0c040fc73883" containerName="istio-proxy" Oct 30 18:23:50.250844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250386 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0634c2b-f6d4-4d94-88d3-8afb1b76b3d0" containerName="istio-proxy" Oct 30 18:23:50.250844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250393 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="31f08f06-6d9f-4003-b35a-62de857fbe80" containerName="istio-proxy" Oct 30 18:23:50.250844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250398 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="94b06967-baa7-4770-9632-f30f27f1dcae" containerName="istio-proxy" Oct 30 18:23:50.250844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250408 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8c0e348-2b8d-48e2-9ec0-a21a83db391e" containerName="istio-proxy" Oct 30 18:23:50.250844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250414 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8c0e348-2b8d-48e2-9ec0-a21a83db391e" containerName="istio-proxy" Oct 30 18:23:50.250844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250421 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="0bbd2e9b-4989-46ca-bf9b-afe2ef671726" containerName="istio-proxy" Oct 30 18:23:50.250844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250428 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5923f01-3342-409d-8cc6-49e03be3a4ab" containerName="istio-proxy" Oct 30 18:23:50.250844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.250434 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c7a6701-721d-448e-af9a-10216921bf48" containerName="istio-proxy" Oct 30 18:23:50.258462 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.258435 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt" Oct 30 18:23:50.275249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.275210 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt"] Oct 30 18:23:50.283566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.282047 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-9066aa\"" Oct 30 18:23:50.283566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.282276 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-9066aa-dockercfg-5g854\"" Oct 30 18:23:50.294471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.294439 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gs94t\" (UniqueName: \"kubernetes.io/projected/76395805-b5c2-4292-a77c-7dc14259a794-kube-api-access-gs94t\") pod \"payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt\" (UID: \"76395805-b5c2-4292-a77c-7dc14259a794\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt" Oct 30 18:23:50.294739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.294717 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76395805-b5c2-4292-a77c-7dc14259a794-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt\" (UID: \"76395805-b5c2-4292-a77c-7dc14259a794\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt" Oct 30 18:23:50.743129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.743088 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gs94t\" (UniqueName: \"kubernetes.io/projected/76395805-b5c2-4292-a77c-7dc14259a794-kube-api-access-gs94t\") pod \"payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt\" (UID: \"76395805-b5c2-4292-a77c-7dc14259a794\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt" Oct 30 18:23:50.743429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.743412 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76395805-b5c2-4292-a77c-7dc14259a794-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt\" (UID: \"76395805-b5c2-4292-a77c-7dc14259a794\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt" Oct 30 18:23:50.775677 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.775604 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76395805-b5c2-4292-a77c-7dc14259a794-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt\" (UID: \"76395805-b5c2-4292-a77c-7dc14259a794\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt" Oct 30 18:23:50.800888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.800858 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gs94t\" (UniqueName: \"kubernetes.io/projected/76395805-b5c2-4292-a77c-7dc14259a794-kube-api-access-gs94t\") pod \"payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt\" (UID: \"76395805-b5c2-4292-a77c-7dc14259a794\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt" Oct 30 18:23:50.818534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.818177 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m"] Oct 30 18:23:50.823153 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.823132 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m"] Oct 30 18:23:50.823256 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.823210 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m" Oct 30 18:23:50.877239 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.877213 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt" Oct 30 18:23:50.949803 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.949020 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb3aa87-34a9-42b3-ac1b-86919834f2d9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m\" (UID: \"4bb3aa87-34a9-42b3-ac1b-86919834f2d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m" Oct 30 18:23:50.949803 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:50.949259 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2x5jh\" (UniqueName: \"kubernetes.io/projected/4bb3aa87-34a9-42b3-ac1b-86919834f2d9-kube-api-access-2x5jh\") pod \"payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m\" (UID: \"4bb3aa87-34a9-42b3-ac1b-86919834f2d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m" Oct 30 18:23:51.050971 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:51.050892 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2x5jh\" (UniqueName: \"kubernetes.io/projected/4bb3aa87-34a9-42b3-ac1b-86919834f2d9-kube-api-access-2x5jh\") pod \"payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m\" (UID: \"4bb3aa87-34a9-42b3-ac1b-86919834f2d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m" Oct 30 18:23:51.051118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:51.051094 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb3aa87-34a9-42b3-ac1b-86919834f2d9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m\" (UID: \"4bb3aa87-34a9-42b3-ac1b-86919834f2d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m" Oct 30 18:23:51.051797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:51.051774 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb3aa87-34a9-42b3-ac1b-86919834f2d9-plugins-config-volume\") pod \"payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m\" (UID: \"4bb3aa87-34a9-42b3-ac1b-86919834f2d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m" Oct 30 18:23:51.070638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:51.070615 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2x5jh\" (UniqueName: \"kubernetes.io/projected/4bb3aa87-34a9-42b3-ac1b-86919834f2d9-kube-api-access-2x5jh\") pod \"payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m\" (UID: \"4bb3aa87-34a9-42b3-ac1b-86919834f2d9\") " pod="openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m" Oct 30 18:23:51.139748 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:51.139722 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m" Oct 30 18:23:51.194640 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:23:51.194606 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76395805_b5c2_4292_a77c_7dc14259a794.slice/crio-19a379f5623bc3057e54b5657e55b9557d5f03b6ba983f592d513cdaf8bd8c8b WatchSource:0}: Error finding container 19a379f5623bc3057e54b5657e55b9557d5f03b6ba983f592d513cdaf8bd8c8b: Status 404 returned error can't find the container with id 19a379f5623bc3057e54b5657e55b9557d5f03b6ba983f592d513cdaf8bd8c8b Oct 30 18:23:51.195489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:51.195462 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt"] Oct 30 18:23:51.447548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:51.447468 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m"] Oct 30 18:23:51.448964 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:23:51.448940 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4bb3aa87_34a9_42b3_ac1b_86919834f2d9.slice/crio-09bfc2338b860d049cf55ed708bc1a0e117be900268caefa6e155657def7788d WatchSource:0}: Error finding container 09bfc2338b860d049cf55ed708bc1a0e117be900268caefa6e155657def7788d: Status 404 returned error can't find the container with id 09bfc2338b860d049cf55ed708bc1a0e117be900268caefa6e155657def7788d Oct 30 18:23:51.957864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:51.957829 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt" event={"ID":"76395805-b5c2-4292-a77c-7dc14259a794","Type":"ContainerStarted","Data":"0d3373fc95ed2da292ecdd9e87338ac7cb5206ada5ad06b5f3c7d2ca32c29c37"} Oct 30 18:23:51.958043 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:51.957887 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt" Oct 30 18:23:51.958043 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:51.957903 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt" event={"ID":"76395805-b5c2-4292-a77c-7dc14259a794","Type":"ContainerStarted","Data":"19a379f5623bc3057e54b5657e55b9557d5f03b6ba983f592d513cdaf8bd8c8b"} Oct 30 18:23:51.959818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:51.959788 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m" event={"ID":"4bb3aa87-34a9-42b3-ac1b-86919834f2d9","Type":"ContainerStarted","Data":"3bbfd22855d3c49bc522f04f4ecf11cf93003189c246711e0d340df4ecdd7b6d"} Oct 30 18:23:51.959920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:51.959839 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m" event={"ID":"4bb3aa87-34a9-42b3-ac1b-86919834f2d9","Type":"ContainerStarted","Data":"09bfc2338b860d049cf55ed708bc1a0e117be900268caefa6e155657def7788d"} Oct 30 18:23:51.959920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:51.959870 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m" Oct 30 18:23:51.999877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:51.999812 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt" podStartSLOduration=1.999796296 podStartE2EDuration="1.999796296s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:51.994984117 +0000 UTC m=+1331.311404440" watchObservedRunningTime="2026-10-30 18:23:51.999796296 +0000 UTC m=+1331.316216619" Oct 30 18:23:52.037332 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:52.037281 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m" podStartSLOduration=2.037267164 podStartE2EDuration="2.037267164s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:52.033893234 +0000 UTC m=+1331.350313551" watchObservedRunningTime="2026-10-30 18:23:52.037267164 +0000 UTC m=+1331.353687485" Oct 30 18:23:54.335751 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.335705 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9"] Oct 30 18:23:54.341984 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.341958 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.360969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.360929 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9"] Oct 30 18:23:54.370544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.367651 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-18cd5a-openshift-default-dockercfg-9fjsx\"" Oct 30 18:23:54.415891 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.415860 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-credential-socket\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.416081 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.415911 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-envoy\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.416081 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.416005 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-token\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.416177 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.416087 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6xczq\" (UniqueName: \"kubernetes.io/projected/7dbbd730-9357-4f2f-b2d3-f41c877205e3-kube-api-access-6xczq\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.416226 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.416213 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-podinfo\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.416299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.416286 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-data\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.416390 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.416374 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-workload-socket\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.416445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.416405 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-workload-certs\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.416577 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.416560 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istiod-ca-cert\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.578533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.571743 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-credential-socket\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.578533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.571818 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-envoy\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.578533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.571881 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-token\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.578533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.571946 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6xczq\" (UniqueName: \"kubernetes.io/projected/7dbbd730-9357-4f2f-b2d3-f41c877205e3-kube-api-access-6xczq\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.578533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.572098 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-podinfo\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.578533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.572181 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-data\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.578533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.572263 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-workload-socket\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.578533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.572285 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-workload-certs\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.578533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.572416 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istiod-ca-cert\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.578533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.573399 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istiod-ca-cert\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.601645 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.598528 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-credential-socket\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.601645 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.598841 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-data\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.601645 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.599103 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-workload-socket\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.601645 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.599402 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-workload-certs\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.669552 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.659823 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-envoy\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.728535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.727782 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-podinfo\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.732534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.729473 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-token\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.826588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.824944 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6xczq\" (UniqueName: \"kubernetes.io/projected/7dbbd730-9357-4f2f-b2d3-f41c877205e3-kube-api-access-6xczq\") pod \"e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:54.965129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:54.965038 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:55.600234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:55.600199 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9"] Oct 30 18:23:55.602213 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:23:55.602185 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7dbbd730_9357_4f2f_b2d3_f41c877205e3.slice/crio-506baaa497bea0e354f427c099515a0bd053fa4b230eb67b71e200fc6ca97f9c WatchSource:0}: Error finding container 506baaa497bea0e354f427c099515a0bd053fa4b230eb67b71e200fc6ca97f9c: Status 404 returned error can't find the container with id 506baaa497bea0e354f427c099515a0bd053fa4b230eb67b71e200fc6ca97f9c Oct 30 18:23:55.606050 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:55.606006 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:23:55.606155 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:55.606086 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:23:55.606155 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:55.606116 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:23:56.045546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:56.044822 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" event={"ID":"7dbbd730-9357-4f2f-b2d3-f41c877205e3","Type":"ContainerStarted","Data":"ef915c07cd6bb50f797e5174e691fd7981623d37390a50fc7a2934a335142f26"} Oct 30 18:23:56.045546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:56.044906 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" event={"ID":"7dbbd730-9357-4f2f-b2d3-f41c877205e3","Type":"ContainerStarted","Data":"506baaa497bea0e354f427c099515a0bd053fa4b230eb67b71e200fc6ca97f9c"} Oct 30 18:23:56.965921 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:56.965890 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:56.972160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:56.972137 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:57.028620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:57.028568 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" podStartSLOduration=3.02854944 podStartE2EDuration="3.02854944s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:23:56.140027814 +0000 UTC m=+1335.456448180" watchObservedRunningTime="2026-10-30 18:23:57.02854944 +0000 UTC m=+1336.344969761" Oct 30 18:23:57.036317 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:23:57.036273 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:23:57.052897 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:57.052872 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:23:57.053979 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:23:57.053958 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:24:00.476045 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:00.476012 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7"] Oct 30 18:24:00.480855 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:00.480837 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" Oct 30 18:24:00.499499 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:00.499468 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7"] Oct 30 18:24:00.505641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:00.505616 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-18cd5a\"" Oct 30 18:24:00.547535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:00.547478 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk"] Oct 30 18:24:00.553460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:00.553432 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" Oct 30 18:24:00.570723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:00.570697 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8g84j\" (UniqueName: \"kubernetes.io/projected/365f97d1-14ff-488c-81aa-1d80e81d6de9-kube-api-access-8g84j\") pod \"payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7\" (UID: \"365f97d1-14ff-488c-81aa-1d80e81d6de9\") " pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" Oct 30 18:24:00.570871 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:00.570858 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365f97d1-14ff-488c-81aa-1d80e81d6de9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7\" (UID: \"365f97d1-14ff-488c-81aa-1d80e81d6de9\") " pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" Oct 30 18:24:00.574569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:00.574545 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk"] Oct 30 18:24:00.589393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:00.589368 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-18cd5a-dockercfg-qn6tk\"" Oct 30 18:24:00.703596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:00.703555 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d0f333-465d-4a99-b2af-a1208c0a12ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk\" (UID: \"99d0f333-465d-4a99-b2af-a1208c0a12ef\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" Oct 30 18:24:00.703781 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:00.703616 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365f97d1-14ff-488c-81aa-1d80e81d6de9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7\" (UID: \"365f97d1-14ff-488c-81aa-1d80e81d6de9\") " pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" Oct 30 18:24:00.707537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:00.703962 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bllpj\" (UniqueName: \"kubernetes.io/projected/99d0f333-465d-4a99-b2af-a1208c0a12ef-kube-api-access-bllpj\") pod \"payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk\" (UID: \"99d0f333-465d-4a99-b2af-a1208c0a12ef\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" Oct 30 18:24:00.707537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:00.704060 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8g84j\" (UniqueName: \"kubernetes.io/projected/365f97d1-14ff-488c-81aa-1d80e81d6de9-kube-api-access-8g84j\") pod \"payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7\" (UID: \"365f97d1-14ff-488c-81aa-1d80e81d6de9\") " pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" Oct 30 18:24:00.850961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:00.825015 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bllpj\" (UniqueName: \"kubernetes.io/projected/99d0f333-465d-4a99-b2af-a1208c0a12ef-kube-api-access-bllpj\") pod \"payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk\" (UID: \"99d0f333-465d-4a99-b2af-a1208c0a12ef\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" Oct 30 18:24:00.850961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:00.825260 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d0f333-465d-4a99-b2af-a1208c0a12ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk\" (UID: \"99d0f333-465d-4a99-b2af-a1208c0a12ef\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" Oct 30 18:24:00.850961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:00.826842 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d0f333-465d-4a99-b2af-a1208c0a12ef-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk\" (UID: \"99d0f333-465d-4a99-b2af-a1208c0a12ef\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" Oct 30 18:24:00.914251 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:00.875724 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365f97d1-14ff-488c-81aa-1d80e81d6de9-plugins-config-volume\") pod \"payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7\" (UID: \"365f97d1-14ff-488c-81aa-1d80e81d6de9\") " pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" Oct 30 18:24:01.204578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:01.164122 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8g84j\" (UniqueName: \"kubernetes.io/projected/365f97d1-14ff-488c-81aa-1d80e81d6de9-kube-api-access-8g84j\") pod \"payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7\" (UID: \"365f97d1-14ff-488c-81aa-1d80e81d6de9\") " pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" Oct 30 18:24:01.204578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:01.173187 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bllpj\" (UniqueName: \"kubernetes.io/projected/99d0f333-465d-4a99-b2af-a1208c0a12ef-kube-api-access-bllpj\") pod \"payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk\" (UID: \"99d0f333-465d-4a99-b2af-a1208c0a12ef\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" Oct 30 18:24:01.204578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:01.187984 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" Oct 30 18:24:01.402091 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:01.401816 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" Oct 30 18:24:02.012427 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:02.012393 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk"] Oct 30 18:24:02.013134 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:24:02.013101 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99d0f333_465d_4a99_b2af_a1208c0a12ef.slice/crio-649942b4f75a25ad6b5b921f27eb1a3e24fd72447f0f34f307afd322b5272f77 WatchSource:0}: Error finding container 649942b4f75a25ad6b5b921f27eb1a3e24fd72447f0f34f307afd322b5272f77: Status 404 returned error can't find the container with id 649942b4f75a25ad6b5b921f27eb1a3e24fd72447f0f34f307afd322b5272f77 Oct 30 18:24:02.079885 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:24:02.079854 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod365f97d1_14ff_488c_81aa_1d80e81d6de9.slice/crio-fa3ba457261ce92283eabc016d583714e496a6c5025ede5c97d0ecf6014a964c WatchSource:0}: Error finding container fa3ba457261ce92283eabc016d583714e496a6c5025ede5c97d0ecf6014a964c: Status 404 returned error can't find the container with id fa3ba457261ce92283eabc016d583714e496a6c5025ede5c97d0ecf6014a964c Oct 30 18:24:02.081215 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:02.080948 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7"] Oct 30 18:24:02.224146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:02.224111 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" event={"ID":"365f97d1-14ff-488c-81aa-1d80e81d6de9","Type":"ContainerStarted","Data":"f3f5eb6672930d3deb1bbb6f94bef96430ac7766049900e2aab8fb29a44f38ea"} Oct 30 18:24:02.224313 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:02.224165 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" event={"ID":"365f97d1-14ff-488c-81aa-1d80e81d6de9","Type":"ContainerStarted","Data":"fa3ba457261ce92283eabc016d583714e496a6c5025ede5c97d0ecf6014a964c"} Oct 30 18:24:02.224313 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:02.224188 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" Oct 30 18:24:02.225879 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:02.225854 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" event={"ID":"99d0f333-465d-4a99-b2af-a1208c0a12ef","Type":"ContainerStarted","Data":"0558ecf38a424601c31bd68cfff7c846cc1a68caf94efdaf3ef6ea188d05ccc9"} Oct 30 18:24:02.225990 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:02.225915 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" event={"ID":"99d0f333-465d-4a99-b2af-a1208c0a12ef","Type":"ContainerStarted","Data":"649942b4f75a25ad6b5b921f27eb1a3e24fd72447f0f34f307afd322b5272f77"} Oct 30 18:24:02.225990 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:02.225950 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" Oct 30 18:24:02.267988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:02.267942 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" podStartSLOduration=2.267928794 podStartE2EDuration="2.267928794s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:02.264166317 +0000 UTC m=+1341.580586652" watchObservedRunningTime="2026-10-30 18:24:02.267928794 +0000 UTC m=+1341.584349114" Oct 30 18:24:02.304136 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:02.304039 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" podStartSLOduration=2.304022654 podStartE2EDuration="2.304022654s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:02.302656801 +0000 UTC m=+1341.619077121" watchObservedRunningTime="2026-10-30 18:24:02.304022654 +0000 UTC m=+1341.620442976" Oct 30 18:24:02.969774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:02.969747 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt" Oct 30 18:24:02.969921 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:02.969815 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m" Oct 30 18:24:03.313053 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.308704 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2"] Oct 30 18:24:03.315289 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.315253 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.321787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.321760 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m"] Oct 30 18:24:03.329592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.328629 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m" Oct 30 18:24:03.336971 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.336944 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w4-c22995-openshift-default-dockercfg-4zn7b\"" Oct 30 18:24:03.380333 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.376563 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2"] Oct 30 18:24:03.380333 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.376925 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:24:03.380551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.377657 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:24:03.383437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.383383 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-workload-socket\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.383703 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.383681 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-envoy\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.383895 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.383878 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe19226-83bf-4dc4-883d-e24502a99599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36a4e9a2-x845m\" (UID: \"cbe19226-83bf-4dc4-883d-e24502a99599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m" Oct 30 18:24:03.384004 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.383990 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-workload-certs\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.384178 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.384164 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-token\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.384276 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.384264 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r8btz\" (UniqueName: \"kubernetes.io/projected/cbe19226-83bf-4dc4-883d-e24502a99599-kube-api-access-r8btz\") pod \"maas-api-revoke-sub-36a4e9a2-x845m\" (UID: \"cbe19226-83bf-4dc4-883d-e24502a99599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m" Oct 30 18:24:03.384403 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.384390 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc2z6\" (UniqueName: \"kubernetes.io/projected/7b16e19a-e62c-47b4-b695-ccc7164524b5-kube-api-access-tc2z6\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.384561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.384547 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-data\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.384677 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.384664 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe19226-83bf-4dc4-883d-e24502a99599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36a4e9a2-x845m\" (UID: \"cbe19226-83bf-4dc4-883d-e24502a99599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m" Oct 30 18:24:03.384807 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.384792 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b16e19a-e62c-47b4-b695-ccc7164524b5-istiod-ca-cert\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.385025 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.385008 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-podinfo\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.385174 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.385159 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-credential-socket\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.391572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.389590 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m"] Oct 30 18:24:03.493611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.487272 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r8btz\" (UniqueName: \"kubernetes.io/projected/cbe19226-83bf-4dc4-883d-e24502a99599-kube-api-access-r8btz\") pod \"maas-api-revoke-sub-36a4e9a2-x845m\" (UID: \"cbe19226-83bf-4dc4-883d-e24502a99599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m" Oct 30 18:24:03.493611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.487347 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc2z6\" (UniqueName: \"kubernetes.io/projected/7b16e19a-e62c-47b4-b695-ccc7164524b5-kube-api-access-tc2z6\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.493611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.487423 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-data\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.493611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.487462 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe19226-83bf-4dc4-883d-e24502a99599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36a4e9a2-x845m\" (UID: \"cbe19226-83bf-4dc4-883d-e24502a99599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m" Oct 30 18:24:03.493611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.487548 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b16e19a-e62c-47b4-b695-ccc7164524b5-istiod-ca-cert\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.493611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.487663 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-podinfo\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.493611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.487710 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-credential-socket\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.493611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.487743 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-workload-socket\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.493611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.487803 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-envoy\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.493611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.487911 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe19226-83bf-4dc4-883d-e24502a99599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36a4e9a2-x845m\" (UID: \"cbe19226-83bf-4dc4-883d-e24502a99599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m" Oct 30 18:24:03.493611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.487946 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-workload-certs\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.493611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.488064 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-token\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.493611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.489196 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-data\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.493611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.489777 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-workload-socket\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.493611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.490357 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b16e19a-e62c-47b4-b695-ccc7164524b5-istiod-ca-cert\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.493611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.491408 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe19226-83bf-4dc4-883d-e24502a99599-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-36a4e9a2-x845m\" (UID: \"cbe19226-83bf-4dc4-883d-e24502a99599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m" Oct 30 18:24:03.493611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.491963 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-workload-certs\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.493611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.492188 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-credential-socket\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.614651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.594403 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-token\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.623541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.616121 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r8btz\" (UniqueName: \"kubernetes.io/projected/cbe19226-83bf-4dc4-883d-e24502a99599-kube-api-access-r8btz\") pod \"maas-api-revoke-sub-36a4e9a2-x845m\" (UID: \"cbe19226-83bf-4dc4-883d-e24502a99599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m" Oct 30 18:24:03.647759 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.639384 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-podinfo\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.647759 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.639683 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-envoy\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.647759 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.639914 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe19226-83bf-4dc4-883d-e24502a99599-maas-api-xks-ca\") pod \"maas-api-revoke-sub-36a4e9a2-x845m\" (UID: \"cbe19226-83bf-4dc4-883d-e24502a99599\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m" Oct 30 18:24:03.659537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.649836 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc2z6\" (UniqueName: \"kubernetes.io/projected/7b16e19a-e62c-47b4-b695-ccc7164524b5-kube-api-access-tc2z6\") pod \"e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:03.696569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.695854 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m" Oct 30 18:24:03.944564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:03.944463 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:04.390200 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:04.390164 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m"] Oct 30 18:24:04.391808 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:24:04.391773 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbe19226_83bf_4dc4_883d_e24502a99599.slice/crio-d159e8371aa2890d8953889fe1b311fef010ed6881953989f8ffa99b1a4b47df WatchSource:0}: Error finding container d159e8371aa2890d8953889fe1b311fef010ed6881953989f8ffa99b1a4b47df: Status 404 returned error can't find the container with id d159e8371aa2890d8953889fe1b311fef010ed6881953989f8ffa99b1a4b47df Oct 30 18:24:04.445901 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:04.445873 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2"] Oct 30 18:24:04.447139 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:24:04.447109 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b16e19a_e62c_47b4_b695_ccc7164524b5.slice/crio-6a824ebad85a21e9311e7f5658ac542e571ae1756d9feebd6839e7a2948c704f WatchSource:0}: Error finding container 6a824ebad85a21e9311e7f5658ac542e571ae1756d9feebd6839e7a2948c704f: Status 404 returned error can't find the container with id 6a824ebad85a21e9311e7f5658ac542e571ae1756d9feebd6839e7a2948c704f Oct 30 18:24:04.449922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:04.449889 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:04.450000 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:04.449954 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:04.450000 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:04.449983 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:05.303803 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:05.303761 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" event={"ID":"7b16e19a-e62c-47b4-b695-ccc7164524b5","Type":"ContainerStarted","Data":"dc88d7c05e156b2e31e11cdf0e8850164c98e39a3b42d279cb811095a855ec7d"} Oct 30 18:24:05.304016 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:05.303860 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" event={"ID":"7b16e19a-e62c-47b4-b695-ccc7164524b5","Type":"ContainerStarted","Data":"6a824ebad85a21e9311e7f5658ac542e571ae1756d9feebd6839e7a2948c704f"} Oct 30 18:24:05.305823 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:05.305794 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m" event={"ID":"cbe19226-83bf-4dc4-883d-e24502a99599","Type":"ContainerStarted","Data":"1c157233554613b7bf37167a4c343225b018f91b61688dd4bad6c683f2690c49"} Oct 30 18:24:05.305952 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:05.305848 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m" event={"ID":"cbe19226-83bf-4dc4-883d-e24502a99599","Type":"ContainerStarted","Data":"d159e8371aa2890d8953889fe1b311fef010ed6881953989f8ffa99b1a4b47df"} Oct 30 18:24:05.372677 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:05.372626 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" podStartSLOduration=2.3726122370000002 podStartE2EDuration="2.372612237s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:05.358038431 +0000 UTC m=+1344.674458754" watchObservedRunningTime="2026-10-30 18:24:05.372612237 +0000 UTC m=+1344.689032570" Oct 30 18:24:05.396109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:05.396063 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m" podStartSLOduration=2.396051615 podStartE2EDuration="2.396051615s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:05.395557648 +0000 UTC m=+1344.711977971" watchObservedRunningTime="2026-10-30 18:24:05.396051615 +0000 UTC m=+1344.712471952" Oct 30 18:24:05.946690 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:05.946655 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:05.952879 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:05.952813 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:06.315498 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:06.315464 5232 generic.go:356] "Generic (PLEG): container finished" podID="cbe19226-83bf-4dc4-883d-e24502a99599" containerID="1c157233554613b7bf37167a4c343225b018f91b61688dd4bad6c683f2690c49" exitCode=0 Oct 30 18:24:06.315721 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:06.315581 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m" event={"ID":"cbe19226-83bf-4dc4-883d-e24502a99599","Type":"ContainerDied","Data":"1c157233554613b7bf37167a4c343225b018f91b61688dd4bad6c683f2690c49"} Oct 30 18:24:06.315871 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:06.315850 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:06.316660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:06.316639 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:24:07.491089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:07.491062 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m" Oct 30 18:24:07.607945 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:07.607863 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cbe19226-83bf-4dc4-883d-e24502a99599-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe19226-83bf-4dc4-883d-e24502a99599-maas-api-xks-ca\") pod \"cbe19226-83bf-4dc4-883d-e24502a99599\" (UID: \"cbe19226-83bf-4dc4-883d-e24502a99599\") " Oct 30 18:24:07.608118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:07.607958 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cbe19226-83bf-4dc4-883d-e24502a99599-kube-api-access-r8btz\" (UniqueName: \"kubernetes.io/projected/cbe19226-83bf-4dc4-883d-e24502a99599-kube-api-access-r8btz\") pod \"cbe19226-83bf-4dc4-883d-e24502a99599\" (UID: \"cbe19226-83bf-4dc4-883d-e24502a99599\") " Oct 30 18:24:07.608118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:07.608041 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cbe19226-83bf-4dc4-883d-e24502a99599-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe19226-83bf-4dc4-883d-e24502a99599-maas-api-openshift-ca\") pod \"cbe19226-83bf-4dc4-883d-e24502a99599\" (UID: \"cbe19226-83bf-4dc4-883d-e24502a99599\") " Oct 30 18:24:07.608468 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:07.608442 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbe19226-83bf-4dc4-883d-e24502a99599-maas-api-openshift-ca" pod "cbe19226-83bf-4dc4-883d-e24502a99599" (UID: "cbe19226-83bf-4dc4-883d-e24502a99599"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:07.608894 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:07.608875 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cbe19226-83bf-4dc4-883d-e24502a99599-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:07.610095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:07.610070 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cbe19226-83bf-4dc4-883d-e24502a99599-maas-api-xks-ca" pod "cbe19226-83bf-4dc4-883d-e24502a99599" (UID: "cbe19226-83bf-4dc4-883d-e24502a99599"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:07.610190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:07.610137 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbe19226-83bf-4dc4-883d-e24502a99599-kube-api-access-r8btz" pod "cbe19226-83bf-4dc4-883d-e24502a99599" (UID: "cbe19226-83bf-4dc4-883d-e24502a99599"). InnerVolumeSpecName "kube-api-access-r8btz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:07.710631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:07.710591 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cbe19226-83bf-4dc4-883d-e24502a99599-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:07.710631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:07.710627 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r8btz\" (UniqueName: \"kubernetes.io/projected/cbe19226-83bf-4dc4-883d-e24502a99599-kube-api-access-r8btz\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:08.334712 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:08.334680 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m" Oct 30 18:24:08.334910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:08.334714 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m" event={"ID":"cbe19226-83bf-4dc4-883d-e24502a99599","Type":"ContainerDied","Data":"d159e8371aa2890d8953889fe1b311fef010ed6881953989f8ffa99b1a4b47df"} Oct 30 18:24:08.334910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:08.334780 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d159e8371aa2890d8953889fe1b311fef010ed6881953989f8ffa99b1a4b47df" Oct 30 18:24:13.269534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:13.268316 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" Oct 30 18:24:13.277786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:13.277757 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" Oct 30 18:24:17.707548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.706096 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9"] Oct 30 18:24:17.710546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.709175 5232 cpu_manager.go:414] "containerMap: removing container" podUID="cbe19226-83bf-4dc4-883d-e24502a99599" containerName="revoke-keys" Oct 30 18:24:17.710546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.709204 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cbe19226-83bf-4dc4-883d-e24502a99599" containerName="revoke-keys" Oct 30 18:24:17.710546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.709401 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="cbe19226-83bf-4dc4-883d-e24502a99599" containerName="revoke-keys" Oct 30 18:24:17.765687 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.765655 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.827986 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.827956 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n"] Oct 30 18:24:17.841338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.841292 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-9af5623f-openshift-default-dockercfg-vxdf5\"" Oct 30 18:24:17.842562 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.842225 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9"] Oct 30 18:24:17.843081 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.843064 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n"] Oct 30 18:24:17.843236 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.842825 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.854301 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.854134 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.854301 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.854214 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.854301 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.854259 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rm79\" (UniqueName: \"kubernetes.io/projected/c4223b02-22ba-4893-a7ea-38dd0d712a9a-kube-api-access-2rm79\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.854484 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.854313 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mvs6\" (UniqueName: \"kubernetes.io/projected/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-kube-api-access-5mvs6\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.854484 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.854381 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.854624 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.854541 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.854731 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.854713 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.854785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.854764 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.854872 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.854852 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.854913 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.854881 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.854948 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.854916 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.854981 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.854966 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.855015 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.854990 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.855049 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.855014 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.855049 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.855039 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.855111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.855079 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.855111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.855093 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.855111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.855108 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.956989 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.956965 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.957156 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.956996 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rm79\" (UniqueName: \"kubernetes.io/projected/c4223b02-22ba-4893-a7ea-38dd0d712a9a-kube-api-access-2rm79\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.957156 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957031 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mvs6\" (UniqueName: \"kubernetes.io/projected/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-kube-api-access-5mvs6\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.957156 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957079 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.957156 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957127 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.957382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957287 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.957382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957341 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.957487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957384 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.957487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957414 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.957642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957568 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.957710 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957675 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.957710 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957679 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.957710 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957576 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.957871 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957719 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.957871 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957755 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.957871 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957759 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.957871 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957784 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.957871 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957824 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.957871 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957832 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.958185 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957909 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.958185 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957919 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.958185 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957948 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.958185 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957969 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.958185 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.957976 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.958185 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.958055 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.958557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.958194 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.958557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.958222 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.958557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.958335 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.960191 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.960163 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:17.960646 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.960624 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.960753 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.960628 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:17.960753 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:17.960688 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:18.005582 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:18.005494 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:18.005818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:18.005722 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mvs6\" (UniqueName: \"kubernetes.io/projected/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-kube-api-access-5mvs6\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:18.006487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:18.006454 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rm79\" (UniqueName: \"kubernetes.io/projected/c4223b02-22ba-4893-a7ea-38dd0d712a9a-kube-api-access-2rm79\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:18.008459 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:18.008431 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:18.097320 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:18.097290 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:18.162726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:18.162408 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:18.508146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:18.508096 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:18.508304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:18.508201 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:18.508304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:18.508246 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:18.515033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:18.514534 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9"] Oct 30 18:24:18.568928 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:18.557568 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n"] Oct 30 18:24:18.590001 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:18.589962 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:18.590135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:18.590040 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:18.590135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:18.590071 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:19.528524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:19.528481 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" event={"ID":"c4223b02-22ba-4893-a7ea-38dd0d712a9a","Type":"ContainerStarted","Data":"e0aa7425dbd1d5ab06058423414ddc13e5760d0762cf4367ec7a4288c5666d94"} Oct 30 18:24:19.529037 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:19.528564 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" event={"ID":"c4223b02-22ba-4893-a7ea-38dd0d712a9a","Type":"ContainerStarted","Data":"678c5f8bfc196e113288a23fa792d5b46cc2bbde86066d50ac6b4906704bb8a1"} Oct 30 18:24:19.530474 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:19.530443 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" event={"ID":"7623b1d3-8da3-4971-b1ba-486f5e1f8db2","Type":"ContainerStarted","Data":"7ef1400449bd017435f6b7ce923a64cb64a6145efafc6015ec46f4e75e959766"} Oct 30 18:24:19.530611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:19.530498 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" event={"ID":"7623b1d3-8da3-4971-b1ba-486f5e1f8db2","Type":"ContainerStarted","Data":"896dbfa3125725bbf63b5cffe55ea45a71b3bf9698632edbbb07f983bad0a4b7"} Oct 30 18:24:19.636068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:19.636012 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podStartSLOduration=2.635988983 podStartE2EDuration="2.635988983s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:19.63540032 +0000 UTC m=+1358.951820640" watchObservedRunningTime="2026-10-30 18:24:19.635988983 +0000 UTC m=+1358.952409348" Oct 30 18:24:19.639223 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:19.639181 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podStartSLOduration=2.6391720100000002 podStartE2EDuration="2.63917201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:19.594651653 +0000 UTC m=+1358.911071974" watchObservedRunningTime="2026-10-30 18:24:19.63917201 +0000 UTC m=+1358.955592331" Oct 30 18:24:20.099015 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:20.098788 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:20.117533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:20.109295 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:20.117533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:20.109376 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:20.163836 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:20.163801 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:20.165766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:20.165737 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:24:20.165900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:20.165795 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:24:21.097713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:21.097633 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:21.097713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:21.097682 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:21.163706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:21.163666 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:24:21.163858 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:21.163716 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:24:22.098355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:22.098320 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:22.098806 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:22.098372 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:22.164007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:22.163973 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:24:22.164159 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:22.164026 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:24:23.097864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:23.097823 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:23.098041 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:23.097886 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:23.163788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:23.163753 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:24:23.164145 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:23.163801 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:24:24.098079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:24.098040 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:24.098285 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:24.098111 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:24.163585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:24.163553 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:24:24.163723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:24.163605 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:24:25.109549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:25.107549 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:25.109549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:25.107628 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:25.164537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:25.164013 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:24:25.164537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:25.164077 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:24:25.712007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:25.711974 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4"] Oct 30 18:24:25.712205 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:25.712183 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" podUID="bae13a01-7f20-4354-8771-6d65d6695651" containerName="istio-proxy" containerID="cri-o://4affc61103204172c0d502385caa862c21fdeeb0804a5953e7dadf8ebea37875" gracePeriod=30 Oct 30 18:24:25.745306 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:25.744829 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2"] Oct 30 18:24:25.745306 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:25.745100 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" podUID="45299d3c-52f3-4d0c-8dd3-af63948cf587" containerName="istio-proxy" containerID="cri-o://66335af6ba8311ec95c88d451a1a62cfa8c560aa3d6585bcc70d99e49ca27869" gracePeriod=30 Oct 30 18:24:25.753260 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:25.753226 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm"] Oct 30 18:24:25.753744 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:25.753642 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" podUID="a337e101-6a41-4416-9731-f28cf2afd0f2" containerName="istio-proxy" containerID="cri-o://c071be777aa99da8d1f7fc20a696da637a9d2e2b43b779f2e7d1dd3d10047766" gracePeriod=30 Oct 30 18:24:25.763302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:25.763266 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj"] Oct 30 18:24:25.763603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:25.763570 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" podUID="03bc2ff1-ddb6-41cf-8238-4df762a46c0e" containerName="istio-proxy" containerID="cri-o://8ef32012e3b73a4bf374336fbd318d9733b1ab44fcc489310ff6d813e835c031" gracePeriod=30 Oct 30 18:24:26.098701 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:26.098668 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:26.098859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:26.098718 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:26.163846 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:26.163814 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:24:26.164321 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:26.163871 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:24:26.677662 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:26.677630 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w"] Oct 30 18:24:26.677880 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:26.677849 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" podUID="5d5ba31c-c8f3-4467-be05-fd146702503e" containerName="istio-proxy" containerID="cri-o://f2171dbcbfc5785dd2caf0edf1669bde96f145454078c94b61223c8e2d73fc95" gracePeriod=30 Oct 30 18:24:27.100066 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.099700 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:27.100066 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.099769 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:27.118968 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:24:27.118925 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:24:27.163374 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.163337 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:24:27.163609 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.163392 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:24:27.494120 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.494094 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb"] Oct 30 18:24:27.499962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.499939 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.518767 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.518731 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-550276-openshift-default-dockercfg-95hsm\"" Oct 30 18:24:27.524108 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.523976 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb"] Oct 30 18:24:27.566094 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.566063 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-token\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.566278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.566125 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43d92d4d-f62f-4b68-a9bf-89186482a244-istiod-ca-cert\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.566278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.566248 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-workload-socket\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.566407 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.566349 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-data\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.566446 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.566421 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-credential-socket\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.566501 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.566454 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btsvg\" (UniqueName: \"kubernetes.io/projected/43d92d4d-f62f-4b68-a9bf-89186482a244-kube-api-access-btsvg\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.566599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.566587 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-workload-certs\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.566697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.566684 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-envoy\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.566752 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.566741 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-podinfo\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.707431 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.707393 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btsvg\" (UniqueName: \"kubernetes.io/projected/43d92d4d-f62f-4b68-a9bf-89186482a244-kube-api-access-btsvg\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.707854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.707829 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-workload-certs\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.708099 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.708078 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-envoy\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.708288 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.708270 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-podinfo\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.708694 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.708672 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-token\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.708927 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.708908 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43d92d4d-f62f-4b68-a9bf-89186482a244-istiod-ca-cert\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.709226 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.709199 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-workload-socket\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.709417 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.709397 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-data\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.709641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.709621 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-credential-socket\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.723537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.710099 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43d92d4d-f62f-4b68-a9bf-89186482a244-istiod-ca-cert\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.723537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.708273 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-workload-certs\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.724479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.724447 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-workload-socket\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.724980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.724955 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-data\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.725350 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.725332 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-credential-socket\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.753839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.753460 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-podinfo\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.765860 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.765758 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btsvg\" (UniqueName: \"kubernetes.io/projected/43d92d4d-f62f-4b68-a9bf-89186482a244-kube-api-access-btsvg\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.768107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.768067 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-envoy\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.769340 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.769312 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-token\") pod \"e2e-api-b-550276-openshift-default-5496d6dbc7-45knb\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:27.822467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:27.822440 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:28.144526 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:28.142707 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:28.144526 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:28.142750 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:28.163494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:28.163158 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:24:28.163494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:28.163213 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:24:28.384255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:28.384100 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb"] Oct 30 18:24:28.387099 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:24:28.387073 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43d92d4d_f62f_4b68_a9bf_89186482a244.slice/crio-07dff41b6e41cc391ad7f9b01afc715f517733acacc2bdc8448816b2f7688605 WatchSource:0}: Error finding container 07dff41b6e41cc391ad7f9b01afc715f517733acacc2bdc8448816b2f7688605: Status 404 returned error can't find the container with id 07dff41b6e41cc391ad7f9b01afc715f517733acacc2bdc8448816b2f7688605 Oct 30 18:24:28.390156 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:28.390097 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:28.390262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:28.390191 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:28.390262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:28.390223 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:28.739607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:28.739442 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" event={"ID":"43d92d4d-f62f-4b68-a9bf-89186482a244","Type":"ContainerStarted","Data":"13d441f3a5c46ec1e0f3f3e38fc0f8692d40a5ce5bcd6d95ba64eeb47729137d"} Oct 30 18:24:28.739607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:28.739607 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" event={"ID":"43d92d4d-f62f-4b68-a9bf-89186482a244","Type":"ContainerStarted","Data":"07dff41b6e41cc391ad7f9b01afc715f517733acacc2bdc8448816b2f7688605"} Oct 30 18:24:28.858535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:28.839179 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" podStartSLOduration=1.839155705 podStartE2EDuration="1.839155705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:28.795389692 +0000 UTC m=+1368.111810014" watchObservedRunningTime="2026-10-30 18:24:28.839155705 +0000 UTC m=+1368.155576027" Oct 30 18:24:28.873530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:28.868132 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:29.111599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:29.099962 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:29.111599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:29.100020 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:29.164542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:29.164375 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:24:29.164542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:29.164435 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:24:29.829590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:29.829561 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:30.098402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:30.098317 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:30.098402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:30.098367 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:30.163323 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:30.163291 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:24:30.163473 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:30.163351 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:24:30.756583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:30.756548 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:30.757574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:30.757549 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:24:31.113308 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.100574 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:31.113308 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.100652 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:31.134057 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.134024 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:24:31.165364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.163780 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:24:31.165364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.163854 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:24:31.175457 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.170209 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:24:31.242645 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.242611 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:24:31.242818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.242670 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" podUID="5d5ba31c-c8f3-4467-be05-fd146702503e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:24:31.252078 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.252048 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.256010 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-envoy\") pod \"45299d3c-52f3-4d0c-8dd3-af63948cf587\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.256062 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-podinfo\") pod \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.256095 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45299d3c-52f3-4d0c-8dd3-af63948cf587-kube-api-access-tg85b\" (UniqueName: \"kubernetes.io/projected/45299d3c-52f3-4d0c-8dd3-af63948cf587-kube-api-access-tg85b\") pod \"45299d3c-52f3-4d0c-8dd3-af63948cf587\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.256120 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istiod-ca-cert\") pod \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.256276 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-envoy\") pod \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.256318 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-token\" (UniqueName: \"kubernetes.io/projected/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-token\") pod \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.256421 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-kube-api-access-zktb7\" (UniqueName: \"kubernetes.io/projected/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-kube-api-access-zktb7\") pod \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.256453 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-podinfo\") pod \"45299d3c-52f3-4d0c-8dd3-af63948cf587\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.256584 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45299d3c-52f3-4d0c-8dd3-af63948cf587-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45299d3c-52f3-4d0c-8dd3-af63948cf587-istiod-ca-cert\") pod \"45299d3c-52f3-4d0c-8dd3-af63948cf587\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.256614 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-workload-socket\") pod \"45299d3c-52f3-4d0c-8dd3-af63948cf587\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.256642 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-data\") pod \"45299d3c-52f3-4d0c-8dd3-af63948cf587\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.256668 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-workload-certs\") pod \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.256767 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-workload-certs\") pod \"45299d3c-52f3-4d0c-8dd3-af63948cf587\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.256750 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istiod-ca-cert" pod "03bc2ff1-ddb6-41cf-8238-4df762a46c0e" (UID: "03bc2ff1-ddb6-41cf-8238-4df762a46c0e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.256859 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-workload-socket\") pod \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.256903 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-data\") pod \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.256988 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-credential-socket\") pod \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\" (UID: \"03bc2ff1-ddb6-41cf-8238-4df762a46c0e\") " Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.257040 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-token\" (UniqueName: \"kubernetes.io/projected/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-token\") pod \"45299d3c-52f3-4d0c-8dd3-af63948cf587\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " Oct 30 18:24:31.257233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.257200 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-data" pod "03bc2ff1-ddb6-41cf-8238-4df762a46c0e" (UID: "03bc2ff1-ddb6-41cf-8238-4df762a46c0e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.258387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.257315 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-credential-socket\") pod \"45299d3c-52f3-4d0c-8dd3-af63948cf587\" (UID: \"45299d3c-52f3-4d0c-8dd3-af63948cf587\") " Oct 30 18:24:31.258387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.258297 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.258387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.258320 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.258634 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.258603 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-credential-socket" pod "45299d3c-52f3-4d0c-8dd3-af63948cf587" (UID: "45299d3c-52f3-4d0c-8dd3-af63948cf587"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.259146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.258893 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-podinfo" pod "03bc2ff1-ddb6-41cf-8238-4df762a46c0e" (UID: "03bc2ff1-ddb6-41cf-8238-4df762a46c0e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:31.260340 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.259901 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-credential-socket" pod "03bc2ff1-ddb6-41cf-8238-4df762a46c0e" (UID: "03bc2ff1-ddb6-41cf-8238-4df762a46c0e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.260340 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.260093 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-workload-socket" pod "45299d3c-52f3-4d0c-8dd3-af63948cf587" (UID: "45299d3c-52f3-4d0c-8dd3-af63948cf587"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.260932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.260747 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-workload-certs" pod "03bc2ff1-ddb6-41cf-8238-4df762a46c0e" (UID: "03bc2ff1-ddb6-41cf-8238-4df762a46c0e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.261240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.261070 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-data" pod "45299d3c-52f3-4d0c-8dd3-af63948cf587" (UID: "45299d3c-52f3-4d0c-8dd3-af63948cf587"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.261240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.261173 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-envoy" pod "03bc2ff1-ddb6-41cf-8238-4df762a46c0e" (UID: "03bc2ff1-ddb6-41cf-8238-4df762a46c0e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.261392 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.261318 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-workload-socket" pod "03bc2ff1-ddb6-41cf-8238-4df762a46c0e" (UID: "03bc2ff1-ddb6-41cf-8238-4df762a46c0e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.261704 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.261489 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-workload-certs" pod "45299d3c-52f3-4d0c-8dd3-af63948cf587" (UID: "45299d3c-52f3-4d0c-8dd3-af63948cf587"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.262022 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.261959 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-envoy" pod "45299d3c-52f3-4d0c-8dd3-af63948cf587" (UID: "45299d3c-52f3-4d0c-8dd3-af63948cf587"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.262022 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.261975 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45299d3c-52f3-4d0c-8dd3-af63948cf587-istiod-ca-cert" pod "45299d3c-52f3-4d0c-8dd3-af63948cf587" (UID: "45299d3c-52f3-4d0c-8dd3-af63948cf587"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:31.264730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.264704 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-kube-api-access-zktb7" pod "03bc2ff1-ddb6-41cf-8238-4df762a46c0e" (UID: "03bc2ff1-ddb6-41cf-8238-4df762a46c0e"). InnerVolumeSpecName "kube-api-access-zktb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:31.264833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.264747 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45299d3c-52f3-4d0c-8dd3-af63948cf587-kube-api-access-tg85b" pod "45299d3c-52f3-4d0c-8dd3-af63948cf587" (UID: "45299d3c-52f3-4d0c-8dd3-af63948cf587"). InnerVolumeSpecName "kube-api-access-tg85b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:31.264833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.264795 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-token" pod "45299d3c-52f3-4d0c-8dd3-af63948cf587" (UID: "45299d3c-52f3-4d0c-8dd3-af63948cf587"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:31.265008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.264988 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-token" pod "03bc2ff1-ddb6-41cf-8238-4df762a46c0e" (UID: "03bc2ff1-ddb6-41cf-8238-4df762a46c0e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:31.265289 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.265171 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-podinfo" pod "45299d3c-52f3-4d0c-8dd3-af63948cf587" (UID: "45299d3c-52f3-4d0c-8dd3-af63948cf587"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:31.277547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.277524 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:24:31.359316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.359229 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-workload-certs\") pod \"bae13a01-7f20-4354-8771-6d65d6695651\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " Oct 30 18:24:31.359316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.359307 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-token\" (UniqueName: \"kubernetes.io/projected/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-token\") pod \"a337e101-6a41-4416-9731-f28cf2afd0f2\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " Oct 30 18:24:31.359578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.359461 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-data\") pod \"a337e101-6a41-4416-9731-f28cf2afd0f2\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " Oct 30 18:24:31.359578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.359525 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-credential-socket\") pod \"bae13a01-7f20-4354-8771-6d65d6695651\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " Oct 30 18:24:31.359578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.359544 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-workload-certs" pod "bae13a01-7f20-4354-8771-6d65d6695651" (UID: "bae13a01-7f20-4354-8771-6d65d6695651"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.359578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.359563 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-workload-certs\") pod \"a337e101-6a41-4416-9731-f28cf2afd0f2\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " Oct 30 18:24:31.359802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.359633 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-workload-socket\") pod \"a337e101-6a41-4416-9731-f28cf2afd0f2\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " Oct 30 18:24:31.359802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.359675 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-envoy\") pod \"a337e101-6a41-4416-9731-f28cf2afd0f2\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " Oct 30 18:24:31.359802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.359709 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a337e101-6a41-4416-9731-f28cf2afd0f2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a337e101-6a41-4416-9731-f28cf2afd0f2-istiod-ca-cert\") pod \"a337e101-6a41-4416-9731-f28cf2afd0f2\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " Oct 30 18:24:31.359802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.359774 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-credential-socket\") pod \"a337e101-6a41-4416-9731-f28cf2afd0f2\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " Oct 30 18:24:31.360011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.359815 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-podinfo\") pod \"a337e101-6a41-4416-9731-f28cf2afd0f2\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " Oct 30 18:24:31.360011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.359827 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-data" pod "a337e101-6a41-4416-9731-f28cf2afd0f2" (UID: "a337e101-6a41-4416-9731-f28cf2afd0f2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.360011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.359850 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-istio-envoy\") pod \"bae13a01-7f20-4354-8771-6d65d6695651\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " Oct 30 18:24:31.360011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.359932 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bae13a01-7f20-4354-8771-6d65d6695651-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae13a01-7f20-4354-8771-6d65d6695651-istio-podinfo\") pod \"bae13a01-7f20-4354-8771-6d65d6695651\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " Oct 30 18:24:31.360011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.359967 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-istio-data\") pod \"bae13a01-7f20-4354-8771-6d65d6695651\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " Oct 30 18:24:31.360253 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.360044 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae13a01-7f20-4354-8771-6d65d6695651-istio-token\" (UniqueName: \"kubernetes.io/projected/bae13a01-7f20-4354-8771-6d65d6695651-istio-token\") pod \"bae13a01-7f20-4354-8771-6d65d6695651\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " Oct 30 18:24:31.360253 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.360091 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bae13a01-7f20-4354-8771-6d65d6695651-kube-api-access-mvdpm\" (UniqueName: \"kubernetes.io/projected/bae13a01-7f20-4354-8771-6d65d6695651-kube-api-access-mvdpm\") pod \"bae13a01-7f20-4354-8771-6d65d6695651\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " Oct 30 18:24:31.360253 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.360168 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a337e101-6a41-4416-9731-f28cf2afd0f2-kube-api-access-shdvd\" (UniqueName: \"kubernetes.io/projected/a337e101-6a41-4416-9731-f28cf2afd0f2-kube-api-access-shdvd\") pod \"a337e101-6a41-4416-9731-f28cf2afd0f2\" (UID: \"a337e101-6a41-4416-9731-f28cf2afd0f2\") " Oct 30 18:24:31.360253 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.360207 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bae13a01-7f20-4354-8771-6d65d6695651-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae13a01-7f20-4354-8771-6d65d6695651-istiod-ca-cert\") pod \"bae13a01-7f20-4354-8771-6d65d6695651\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " Oct 30 18:24:31.360455 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.360370 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-workload-socket\") pod \"bae13a01-7f20-4354-8771-6d65d6695651\" (UID: \"bae13a01-7f20-4354-8771-6d65d6695651\") " Oct 30 18:24:31.361395 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.361366 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-workload-certs" pod "a337e101-6a41-4416-9731-f28cf2afd0f2" (UID: "a337e101-6a41-4416-9731-f28cf2afd0f2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.361625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.361603 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.361936 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.361691 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-workload-socket" pod "bae13a01-7f20-4354-8771-6d65d6695651" (UID: "bae13a01-7f20-4354-8771-6d65d6695651"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.361936 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.361793 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a337e101-6a41-4416-9731-f28cf2afd0f2-istiod-ca-cert" pod "a337e101-6a41-4416-9731-f28cf2afd0f2" (UID: "a337e101-6a41-4416-9731-f28cf2afd0f2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:31.362158 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.361944 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-credential-socket" pod "a337e101-6a41-4416-9731-f28cf2afd0f2" (UID: "a337e101-6a41-4416-9731-f28cf2afd0f2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.362227 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.361631 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.362227 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.362199 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.362227 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.362216 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.362385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.362230 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.362385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.362242 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.362385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.362255 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.362385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.362267 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tg85b\" (UniqueName: \"kubernetes.io/projected/45299d3c-52f3-4d0c-8dd3-af63948cf587-kube-api-access-tg85b\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.362385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.362279 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.362385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.362290 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.362385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.362301 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.362385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.362312 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zktb7\" (UniqueName: \"kubernetes.io/projected/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-kube-api-access-zktb7\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.362385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.362324 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.362385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.362335 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/45299d3c-52f3-4d0c-8dd3-af63948cf587-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.362385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.362347 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.362385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.362346 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-token" pod "a337e101-6a41-4416-9731-f28cf2afd0f2" (UID: "a337e101-6a41-4416-9731-f28cf2afd0f2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:31.362385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.362357 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.362385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.362369 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03bc2ff1-ddb6-41cf-8238-4df762a46c0e-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.362385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.362380 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/45299d3c-52f3-4d0c-8dd3-af63948cf587-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.362385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.362390 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.364830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.363412 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-istio-envoy" pod "bae13a01-7f20-4354-8771-6d65d6695651" (UID: "bae13a01-7f20-4354-8771-6d65d6695651"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.364830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.363753 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-workload-socket" pod "a337e101-6a41-4416-9731-f28cf2afd0f2" (UID: "a337e101-6a41-4416-9731-f28cf2afd0f2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.364830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.363915 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-credential-socket" pod "bae13a01-7f20-4354-8771-6d65d6695651" (UID: "bae13a01-7f20-4354-8771-6d65d6695651"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.364830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.363965 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-istio-data" pod "bae13a01-7f20-4354-8771-6d65d6695651" (UID: "bae13a01-7f20-4354-8771-6d65d6695651"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.364830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.364202 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bae13a01-7f20-4354-8771-6d65d6695651-istiod-ca-cert" pod "bae13a01-7f20-4354-8771-6d65d6695651" (UID: "bae13a01-7f20-4354-8771-6d65d6695651"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:31.365852 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.365819 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a337e101-6a41-4416-9731-f28cf2afd0f2-kube-api-access-shdvd" pod "a337e101-6a41-4416-9731-f28cf2afd0f2" (UID: "a337e101-6a41-4416-9731-f28cf2afd0f2"). InnerVolumeSpecName "kube-api-access-shdvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:31.366098 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.366069 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bae13a01-7f20-4354-8771-6d65d6695651-istio-podinfo" pod "bae13a01-7f20-4354-8771-6d65d6695651" (UID: "bae13a01-7f20-4354-8771-6d65d6695651"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:31.366547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.366498 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae13a01-7f20-4354-8771-6d65d6695651-kube-api-access-mvdpm" pod "bae13a01-7f20-4354-8771-6d65d6695651" (UID: "bae13a01-7f20-4354-8771-6d65d6695651"). InnerVolumeSpecName "kube-api-access-mvdpm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:31.367932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.367901 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-envoy" pod "a337e101-6a41-4416-9731-f28cf2afd0f2" (UID: "a337e101-6a41-4416-9731-f28cf2afd0f2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:31.368093 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.368057 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bae13a01-7f20-4354-8771-6d65d6695651-istio-token" pod "bae13a01-7f20-4354-8771-6d65d6695651" (UID: "bae13a01-7f20-4354-8771-6d65d6695651"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:31.369935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.369910 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-podinfo" pod "a337e101-6a41-4416-9731-f28cf2afd0f2" (UID: "a337e101-6a41-4416-9731-f28cf2afd0f2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:31.469387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.468764 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shdvd\" (UniqueName: \"kubernetes.io/projected/a337e101-6a41-4416-9731-f28cf2afd0f2-kube-api-access-shdvd\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.469387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.468810 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bae13a01-7f20-4354-8771-6d65d6695651-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.469387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.468826 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.469387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.468839 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.469387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.468854 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.469387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.468867 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.469387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.468880 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.469387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.468896 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a337e101-6a41-4416-9731-f28cf2afd0f2-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.469387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.468909 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a337e101-6a41-4416-9731-f28cf2afd0f2-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.469387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.468921 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a337e101-6a41-4416-9731-f28cf2afd0f2-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.469387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.468935 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.469387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.468949 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bae13a01-7f20-4354-8771-6d65d6695651-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.469387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.468961 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bae13a01-7f20-4354-8771-6d65d6695651-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.469387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.468973 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bae13a01-7f20-4354-8771-6d65d6695651-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.469387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.468986 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mvdpm\" (UniqueName: \"kubernetes.io/projected/bae13a01-7f20-4354-8771-6d65d6695651-kube-api-access-mvdpm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:31.768012 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.767971 5232 generic.go:356] "Generic (PLEG): container finished" podID="03bc2ff1-ddb6-41cf-8238-4df762a46c0e" containerID="8ef32012e3b73a4bf374336fbd318d9733b1ab44fcc489310ff6d813e835c031" exitCode=0 Oct 30 18:24:31.768179 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.768033 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" Oct 30 18:24:31.768179 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.768060 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" event={"ID":"03bc2ff1-ddb6-41cf-8238-4df762a46c0e","Type":"ContainerDied","Data":"8ef32012e3b73a4bf374336fbd318d9733b1ab44fcc489310ff6d813e835c031"} Oct 30 18:24:31.768179 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.768139 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj" event={"ID":"03bc2ff1-ddb6-41cf-8238-4df762a46c0e","Type":"ContainerDied","Data":"ac440a882a062dd282671299bfd7e9b6d670729b861394be78f1422e2f4b5ffa"} Oct 30 18:24:31.768179 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.768165 5232 scope.go:122] "RemoveContainer" containerID="8ef32012e3b73a4bf374336fbd318d9733b1ab44fcc489310ff6d813e835c031" Oct 30 18:24:31.770106 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.770074 5232 generic.go:356] "Generic (PLEG): container finished" podID="a337e101-6a41-4416-9731-f28cf2afd0f2" containerID="c071be777aa99da8d1f7fc20a696da637a9d2e2b43b779f2e7d1dd3d10047766" exitCode=0 Oct 30 18:24:31.770238 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.770111 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" event={"ID":"a337e101-6a41-4416-9731-f28cf2afd0f2","Type":"ContainerDied","Data":"c071be777aa99da8d1f7fc20a696da637a9d2e2b43b779f2e7d1dd3d10047766"} Oct 30 18:24:31.770238 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.770172 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" event={"ID":"a337e101-6a41-4416-9731-f28cf2afd0f2","Type":"ContainerDied","Data":"5f12c0230e3b55c321c10f4457e359da2923e4cb26431590b623716eb1afcc3c"} Oct 30 18:24:31.770460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.770436 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm" Oct 30 18:24:31.773097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.772771 5232 generic.go:356] "Generic (PLEG): container finished" podID="bae13a01-7f20-4354-8771-6d65d6695651" containerID="4affc61103204172c0d502385caa862c21fdeeb0804a5953e7dadf8ebea37875" exitCode=0 Oct 30 18:24:31.773097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.772873 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" event={"ID":"bae13a01-7f20-4354-8771-6d65d6695651","Type":"ContainerDied","Data":"4affc61103204172c0d502385caa862c21fdeeb0804a5953e7dadf8ebea37875"} Oct 30 18:24:31.773097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.772911 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" event={"ID":"bae13a01-7f20-4354-8771-6d65d6695651","Type":"ContainerDied","Data":"9de8b21475441cf3c4816f1c7cbb4170a0a4e30fef183525c258d66ab7b0c2ca"} Oct 30 18:24:31.773097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.772972 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4" Oct 30 18:24:31.777325 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.777299 5232 generic.go:356] "Generic (PLEG): container finished" podID="45299d3c-52f3-4d0c-8dd3-af63948cf587" containerID="66335af6ba8311ec95c88d451a1a62cfa8c560aa3d6585bcc70d99e49ca27869" exitCode=0 Oct 30 18:24:31.778066 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.778047 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" Oct 30 18:24:31.782317 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.782292 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" event={"ID":"45299d3c-52f3-4d0c-8dd3-af63948cf587","Type":"ContainerDied","Data":"66335af6ba8311ec95c88d451a1a62cfa8c560aa3d6585bcc70d99e49ca27869"} Oct 30 18:24:31.782754 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.782456 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2" event={"ID":"45299d3c-52f3-4d0c-8dd3-af63948cf587","Type":"ContainerDied","Data":"5804ea074e682766625ec940d6831f6d90be07595591917865df1510c6efa876"} Oct 30 18:24:31.800679 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.800655 5232 scope.go:122] "RemoveContainer" containerID="8ef32012e3b73a4bf374336fbd318d9733b1ab44fcc489310ff6d813e835c031" Oct 30 18:24:31.801026 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:24:31.801006 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ef32012e3b73a4bf374336fbd318d9733b1ab44fcc489310ff6d813e835c031\": container with ID starting with 8ef32012e3b73a4bf374336fbd318d9733b1ab44fcc489310ff6d813e835c031 not found: ID does not exist" containerID="8ef32012e3b73a4bf374336fbd318d9733b1ab44fcc489310ff6d813e835c031" Oct 30 18:24:31.801093 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.801033 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ef32012e3b73a4bf374336fbd318d9733b1ab44fcc489310ff6d813e835c031"} err="failed to get container status \"8ef32012e3b73a4bf374336fbd318d9733b1ab44fcc489310ff6d813e835c031\": rpc error: code = NotFound desc = could not find container \"8ef32012e3b73a4bf374336fbd318d9733b1ab44fcc489310ff6d813e835c031\": container with ID starting with 8ef32012e3b73a4bf374336fbd318d9733b1ab44fcc489310ff6d813e835c031 not found: ID does not exist" Oct 30 18:24:31.801093 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.801050 5232 scope.go:122] "RemoveContainer" containerID="c071be777aa99da8d1f7fc20a696da637a9d2e2b43b779f2e7d1dd3d10047766" Oct 30 18:24:31.815759 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.815738 5232 scope.go:122] "RemoveContainer" containerID="c071be777aa99da8d1f7fc20a696da637a9d2e2b43b779f2e7d1dd3d10047766" Oct 30 18:24:31.816317 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:24:31.816113 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c071be777aa99da8d1f7fc20a696da637a9d2e2b43b779f2e7d1dd3d10047766\": container with ID starting with c071be777aa99da8d1f7fc20a696da637a9d2e2b43b779f2e7d1dd3d10047766 not found: ID does not exist" containerID="c071be777aa99da8d1f7fc20a696da637a9d2e2b43b779f2e7d1dd3d10047766" Oct 30 18:24:31.816317 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.816158 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c071be777aa99da8d1f7fc20a696da637a9d2e2b43b779f2e7d1dd3d10047766"} err="failed to get container status \"c071be777aa99da8d1f7fc20a696da637a9d2e2b43b779f2e7d1dd3d10047766\": rpc error: code = NotFound desc = could not find container \"c071be777aa99da8d1f7fc20a696da637a9d2e2b43b779f2e7d1dd3d10047766\": container with ID starting with c071be777aa99da8d1f7fc20a696da637a9d2e2b43b779f2e7d1dd3d10047766 not found: ID does not exist" Oct 30 18:24:31.816317 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.816183 5232 scope.go:122] "RemoveContainer" containerID="4affc61103204172c0d502385caa862c21fdeeb0804a5953e7dadf8ebea37875" Oct 30 18:24:31.827951 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.827924 5232 scope.go:122] "RemoveContainer" containerID="4affc61103204172c0d502385caa862c21fdeeb0804a5953e7dadf8ebea37875" Oct 30 18:24:31.828234 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:24:31.828212 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4affc61103204172c0d502385caa862c21fdeeb0804a5953e7dadf8ebea37875\": container with ID starting with 4affc61103204172c0d502385caa862c21fdeeb0804a5953e7dadf8ebea37875 not found: ID does not exist" containerID="4affc61103204172c0d502385caa862c21fdeeb0804a5953e7dadf8ebea37875" Oct 30 18:24:31.828323 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.828245 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4affc61103204172c0d502385caa862c21fdeeb0804a5953e7dadf8ebea37875"} err="failed to get container status \"4affc61103204172c0d502385caa862c21fdeeb0804a5953e7dadf8ebea37875\": rpc error: code = NotFound desc = could not find container \"4affc61103204172c0d502385caa862c21fdeeb0804a5953e7dadf8ebea37875\": container with ID starting with 4affc61103204172c0d502385caa862c21fdeeb0804a5953e7dadf8ebea37875 not found: ID does not exist" Oct 30 18:24:31.828323 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.828269 5232 scope.go:122] "RemoveContainer" containerID="66335af6ba8311ec95c88d451a1a62cfa8c560aa3d6585bcc70d99e49ca27869" Oct 30 18:24:31.841768 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.841732 5232 scope.go:122] "RemoveContainer" containerID="66335af6ba8311ec95c88d451a1a62cfa8c560aa3d6585bcc70d99e49ca27869" Oct 30 18:24:31.842164 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:24:31.842132 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66335af6ba8311ec95c88d451a1a62cfa8c560aa3d6585bcc70d99e49ca27869\": container with ID starting with 66335af6ba8311ec95c88d451a1a62cfa8c560aa3d6585bcc70d99e49ca27869 not found: ID does not exist" containerID="66335af6ba8311ec95c88d451a1a62cfa8c560aa3d6585bcc70d99e49ca27869" Oct 30 18:24:31.842276 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.842163 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66335af6ba8311ec95c88d451a1a62cfa8c560aa3d6585bcc70d99e49ca27869"} err="failed to get container status \"66335af6ba8311ec95c88d451a1a62cfa8c560aa3d6585bcc70d99e49ca27869\": rpc error: code = NotFound desc = could not find container \"66335af6ba8311ec95c88d451a1a62cfa8c560aa3d6585bcc70d99e49ca27869\": container with ID starting with 66335af6ba8311ec95c88d451a1a62cfa8c560aa3d6585bcc70d99e49ca27869 not found: ID does not exist" Oct 30 18:24:31.873464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.873420 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm"] Oct 30 18:24:31.885469 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.884560 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-qf5dm"] Oct 30 18:24:31.929062 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.929030 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2"] Oct 30 18:24:31.934170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.934141 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-tl8r2"] Oct 30 18:24:31.966638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.966583 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj"] Oct 30 18:24:31.973410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.973381 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-569dj"] Oct 30 18:24:31.999063 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:31.999040 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:24:32.010161 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.010132 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4"] Oct 30 18:24:32.014213 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.014187 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-fdf4da-openshift-default-76967bcf57-z6rx4"] Oct 30 18:24:32.079065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.079030 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-envoy\") pod \"5d5ba31c-c8f3-4467-be05-fd146702503e\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " Oct 30 18:24:32.079241 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.079135 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-credential-socket\") pod \"5d5ba31c-c8f3-4467-be05-fd146702503e\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " Oct 30 18:24:32.079309 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.079289 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-token\" (UniqueName: \"kubernetes.io/projected/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-token\") pod \"5d5ba31c-c8f3-4467-be05-fd146702503e\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " Oct 30 18:24:32.079424 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.079405 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-workload-socket\") pod \"5d5ba31c-c8f3-4467-be05-fd146702503e\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " Oct 30 18:24:32.079489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.079466 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-credential-socket" pod "5d5ba31c-c8f3-4467-be05-fd146702503e" (UID: "5d5ba31c-c8f3-4467-be05-fd146702503e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:32.079489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.079479 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-data\") pod \"5d5ba31c-c8f3-4467-be05-fd146702503e\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " Oct 30 18:24:32.079632 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.079529 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d5ba31c-c8f3-4467-be05-fd146702503e-kube-api-access-flqgw\" (UniqueName: \"kubernetes.io/projected/5d5ba31c-c8f3-4467-be05-fd146702503e-kube-api-access-flqgw\") pod \"5d5ba31c-c8f3-4467-be05-fd146702503e\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " Oct 30 18:24:32.079632 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.079572 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d5ba31c-c8f3-4467-be05-fd146702503e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5ba31c-c8f3-4467-be05-fd146702503e-istiod-ca-cert\") pod \"5d5ba31c-c8f3-4467-be05-fd146702503e\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " Oct 30 18:24:32.079800 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.079669 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-podinfo\") pod \"5d5ba31c-c8f3-4467-be05-fd146702503e\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " Oct 30 18:24:32.079800 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.079680 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-workload-socket" pod "5d5ba31c-c8f3-4467-be05-fd146702503e" (UID: "5d5ba31c-c8f3-4467-be05-fd146702503e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:32.079800 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.079769 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-data" pod "5d5ba31c-c8f3-4467-be05-fd146702503e" (UID: "5d5ba31c-c8f3-4467-be05-fd146702503e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:32.079800 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.079796 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-workload-certs\") pod \"5d5ba31c-c8f3-4467-be05-fd146702503e\" (UID: \"5d5ba31c-c8f3-4467-be05-fd146702503e\") " Oct 30 18:24:32.080431 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.080292 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-workload-certs" pod "5d5ba31c-c8f3-4467-be05-fd146702503e" (UID: "5d5ba31c-c8f3-4467-be05-fd146702503e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:32.080431 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.080404 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d5ba31c-c8f3-4467-be05-fd146702503e-istiod-ca-cert" pod "5d5ba31c-c8f3-4467-be05-fd146702503e" (UID: "5d5ba31c-c8f3-4467-be05-fd146702503e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:32.080755 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.080735 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:32.080833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.080760 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:32.080833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.080776 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5d5ba31c-c8f3-4467-be05-fd146702503e-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:32.080833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.080789 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:32.080833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.080801 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:32.081966 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.081930 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-envoy" pod "5d5ba31c-c8f3-4467-be05-fd146702503e" (UID: "5d5ba31c-c8f3-4467-be05-fd146702503e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:32.082379 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.082355 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-podinfo" pod "5d5ba31c-c8f3-4467-be05-fd146702503e" (UID: "5d5ba31c-c8f3-4467-be05-fd146702503e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:32.083058 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.083032 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-token" pod "5d5ba31c-c8f3-4467-be05-fd146702503e" (UID: "5d5ba31c-c8f3-4467-be05-fd146702503e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:32.083143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.083093 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d5ba31c-c8f3-4467-be05-fd146702503e-kube-api-access-flqgw" pod "5d5ba31c-c8f3-4467-be05-fd146702503e" (UID: "5d5ba31c-c8f3-4467-be05-fd146702503e"). InnerVolumeSpecName "kube-api-access-flqgw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:32.098256 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.098228 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:32.098340 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.098271 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:32.171471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.163145 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:24:32.171471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.163203 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:24:32.182159 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.182126 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:32.182288 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.182164 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-flqgw\" (UniqueName: \"kubernetes.io/projected/5d5ba31c-c8f3-4467-be05-fd146702503e-kube-api-access-flqgw\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:32.182288 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.182181 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:32.182288 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.182191 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5d5ba31c-c8f3-4467-be05-fd146702503e-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:32.777381 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.777351 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s"] Oct 30 18:24:32.778261 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.778240 5232 cpu_manager.go:414] "containerMap: removing container" podUID="bae13a01-7f20-4354-8771-6d65d6695651" containerName="istio-proxy" Oct 30 18:24:32.778368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.778265 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bae13a01-7f20-4354-8771-6d65d6695651" containerName="istio-proxy" Oct 30 18:24:32.778368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.778276 5232 cpu_manager.go:414] "containerMap: removing container" podUID="a337e101-6a41-4416-9731-f28cf2afd0f2" containerName="istio-proxy" Oct 30 18:24:32.778368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.778282 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a337e101-6a41-4416-9731-f28cf2afd0f2" containerName="istio-proxy" Oct 30 18:24:32.778368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.778297 5232 cpu_manager.go:414] "containerMap: removing container" podUID="5d5ba31c-c8f3-4467-be05-fd146702503e" containerName="istio-proxy" Oct 30 18:24:32.778368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.778302 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d5ba31c-c8f3-4467-be05-fd146702503e" containerName="istio-proxy" Oct 30 18:24:32.778368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.778317 5232 cpu_manager.go:414] "containerMap: removing container" podUID="03bc2ff1-ddb6-41cf-8238-4df762a46c0e" containerName="istio-proxy" Oct 30 18:24:32.778368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.778321 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03bc2ff1-ddb6-41cf-8238-4df762a46c0e" containerName="istio-proxy" Oct 30 18:24:32.778368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.778342 5232 cpu_manager.go:414] "containerMap: removing container" podUID="45299d3c-52f3-4d0c-8dd3-af63948cf587" containerName="istio-proxy" Oct 30 18:24:32.778368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.778348 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45299d3c-52f3-4d0c-8dd3-af63948cf587" containerName="istio-proxy" Oct 30 18:24:32.778870 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.778472 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="03bc2ff1-ddb6-41cf-8238-4df762a46c0e" containerName="istio-proxy" Oct 30 18:24:32.778870 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.778486 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="45299d3c-52f3-4d0c-8dd3-af63948cf587" containerName="istio-proxy" Oct 30 18:24:32.778870 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.778524 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="bae13a01-7f20-4354-8771-6d65d6695651" containerName="istio-proxy" Oct 30 18:24:32.778870 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.778533 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="a337e101-6a41-4416-9731-f28cf2afd0f2" containerName="istio-proxy" Oct 30 18:24:32.778870 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.778544 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d5ba31c-c8f3-4467-be05-fd146702503e" containerName="istio-proxy" Oct 30 18:24:32.800431 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.800394 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:32.831932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.831896 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03bc2ff1-ddb6-41cf-8238-4df762a46c0e" path="/var/lib/kubelet/pods/03bc2ff1-ddb6-41cf-8238-4df762a46c0e/volumes" Oct 30 18:24:32.832992 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.832968 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45299d3c-52f3-4d0c-8dd3-af63948cf587" path="/var/lib/kubelet/pods/45299d3c-52f3-4d0c-8dd3-af63948cf587/volumes" Oct 30 18:24:32.834408 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.834386 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a337e101-6a41-4416-9731-f28cf2afd0f2" path="/var/lib/kubelet/pods/a337e101-6a41-4416-9731-f28cf2afd0f2/volumes" Oct 30 18:24:32.835635 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.835615 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bae13a01-7f20-4354-8771-6d65d6695651" path="/var/lib/kubelet/pods/bae13a01-7f20-4354-8771-6d65d6695651/volumes" Oct 30 18:24:32.836489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.836471 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg"] Oct 30 18:24:32.859777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.859751 5232 generic.go:356] "Generic (PLEG): container finished" podID="5d5ba31c-c8f3-4467-be05-fd146702503e" containerID="f2171dbcbfc5785dd2caf0edf1669bde96f145454078c94b61223c8e2d73fc95" exitCode=0 Oct 30 18:24:32.860076 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.860058 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" Oct 30 18:24:32.860166 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.860094 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n"] Oct 30 18:24:32.860382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.860361 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:32.866082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.866056 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s"] Oct 30 18:24:32.866191 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.866108 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg"] Oct 30 18:24:32.866191 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.866135 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n"] Oct 30 18:24:32.866191 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.866167 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" event={"ID":"5d5ba31c-c8f3-4467-be05-fd146702503e","Type":"ContainerDied","Data":"f2171dbcbfc5785dd2caf0edf1669bde96f145454078c94b61223c8e2d73fc95"} Oct 30 18:24:32.866191 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.866177 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:32.866405 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.866196 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w" event={"ID":"5d5ba31c-c8f3-4467-be05-fd146702503e","Type":"ContainerDied","Data":"d299c1cc392289a4c387ad436ba7a25e9dfd737ecd80419ff6cd44ccfeeff3df"} Oct 30 18:24:32.866405 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.866228 5232 scope.go:122] "RemoveContainer" containerID="f2171dbcbfc5785dd2caf0edf1669bde96f145454078c94b61223c8e2d73fc95" Oct 30 18:24:32.886796 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.886772 5232 scope.go:122] "RemoveContainer" containerID="f2171dbcbfc5785dd2caf0edf1669bde96f145454078c94b61223c8e2d73fc95" Oct 30 18:24:32.887220 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:24:32.887191 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f2171dbcbfc5785dd2caf0edf1669bde96f145454078c94b61223c8e2d73fc95\": container with ID starting with f2171dbcbfc5785dd2caf0edf1669bde96f145454078c94b61223c8e2d73fc95 not found: ID does not exist" containerID="f2171dbcbfc5785dd2caf0edf1669bde96f145454078c94b61223c8e2d73fc95" Oct 30 18:24:32.887332 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.887233 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f2171dbcbfc5785dd2caf0edf1669bde96f145454078c94b61223c8e2d73fc95"} err="failed to get container status \"f2171dbcbfc5785dd2caf0edf1669bde96f145454078c94b61223c8e2d73fc95\": rpc error: code = NotFound desc = could not find container \"f2171dbcbfc5785dd2caf0edf1669bde96f145454078c94b61223c8e2d73fc95\": container with ID starting with f2171dbcbfc5785dd2caf0edf1669bde96f145454078c94b61223c8e2d73fc95 not found: ID does not exist" Oct 30 18:24:32.899465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.899431 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:32.899644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.899625 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:32.899698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.899688 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:32.899816 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.899796 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:32.899876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.899834 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:32.899938 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.899911 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:32.900047 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.900026 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1fd255d-3e8c-489a-9151-cb9533af2405-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:32.900106 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.900091 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:32.900155 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:32.900141 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btkgv\" (UniqueName: \"kubernetes.io/projected/d1fd255d-3e8c-489a-9151-cb9533af2405-kube-api-access-btkgv\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.002165 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002133 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.002165 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002170 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.002413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002191 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.002413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002251 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.002413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002309 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.002413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002378 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.002650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002448 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.002650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002566 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1fd255d-3e8c-489a-9151-cb9533af2405-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.002650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002566 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.002805 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002659 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.002805 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002685 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gczwr\" (UniqueName: \"kubernetes.io/projected/352fefd4-5ce5-4ce2-81ff-28769154e70b-kube-api-access-gczwr\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.002805 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002710 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.002805 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002767 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.002805 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002793 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.003050 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002844 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.003050 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002872 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.003050 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002889 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btkgv\" (UniqueName: \"kubernetes.io/projected/d1fd255d-3e8c-489a-9151-cb9533af2405-kube-api-access-btkgv\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.003050 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002920 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352fefd4-5ce5-4ce2-81ff-28769154e70b-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.003050 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.002940 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.003050 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.003020 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.003450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.003065 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.003450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.003080 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.003450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.003186 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.003450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.003262 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.003450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.003287 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.003450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.003339 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.003450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.003371 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n89s5\" (UniqueName: \"kubernetes.io/projected/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-kube-api-access-n89s5\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.003839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.003713 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.003839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.003763 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.003939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.003882 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.004020 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.004001 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.004131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.004117 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w"] Oct 30 18:24:33.004327 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.004310 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1fd255d-3e8c-489a-9151-cb9533af2405-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.006099 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.006070 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.006422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.006401 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.008458 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.008439 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-9jw4w"] Oct 30 18:24:33.022616 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.022586 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btkgv\" (UniqueName: \"kubernetes.io/projected/d1fd255d-3e8c-489a-9151-cb9533af2405-kube-api-access-btkgv\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.022788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.022773 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.097971 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.097895 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:33.098155 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.098135 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:33.098237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.098190 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:33.105569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.105503 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.105664 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.105584 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gczwr\" (UniqueName: \"kubernetes.io/projected/352fefd4-5ce5-4ce2-81ff-28769154e70b-kube-api-access-gczwr\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.105664 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.105612 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.105664 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.105637 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.105815 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.105678 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.105815 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.105723 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352fefd4-5ce5-4ce2-81ff-28769154e70b-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.105815 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.105753 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.105969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.105874 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.105969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.105948 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.106086 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.106005 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.106086 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.106019 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.106086 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.106080 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.106257 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.106108 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.106257 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.106171 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n89s5\" (UniqueName: \"kubernetes.io/projected/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-kube-api-access-n89s5\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.106257 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.106234 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.106434 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.106269 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.106434 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.106284 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.106571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.106520 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.106629 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.106594 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.106839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.106816 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-credential-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.106839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.106599 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.106990 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.106829 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.106990 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.106908 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-data\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.106990 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.106915 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.106990 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.106925 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352fefd4-5ce5-4ce2-81ff-28769154e70b-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.107310 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.107264 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-workload-socket\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.107575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.107501 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-workload-certs\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.107770 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.107748 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istiod-ca-cert\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.108265 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.108245 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.108794 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.108774 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.109640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.109616 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-podinfo\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.109716 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.109618 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-envoy\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.125879 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.125854 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.125955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.125883 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gczwr\" (UniqueName: \"kubernetes.io/projected/352fefd4-5ce5-4ce2-81ff-28769154e70b-kube-api-access-gczwr\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.126915 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.126888 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n89s5\" (UniqueName: \"kubernetes.io/projected/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-kube-api-access-n89s5\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.127027 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.127011 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-token\") pod \"e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.157864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.157843 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:33.163473 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.163451 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:33.163800 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.163778 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:24:33.163873 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.163829 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:24:33.192450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.192425 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:33.197867 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.197843 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:33.480273 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.480242 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s"] Oct 30 18:24:33.481521 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:24:33.481473 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd1fd255d_3e8c_489a_9151_cb9533af2405.slice/crio-4b1a97298b0a8027e4fcfe091980ed255740466aa32814ca7e7a0b043416c739 WatchSource:0}: Error finding container 4b1a97298b0a8027e4fcfe091980ed255740466aa32814ca7e7a0b043416c739: Status 404 returned error can't find the container with id 4b1a97298b0a8027e4fcfe091980ed255740466aa32814ca7e7a0b043416c739 Oct 30 18:24:33.484639 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.484607 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:33.484725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.484669 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:33.484725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.484701 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:33.530736 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.530711 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg"] Oct 30 18:24:33.533010 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:24:33.532979 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod352fefd4_5ce5_4ce2_81ff_28769154e70b.slice/crio-9e2ac10b0dff16cf351ca51062109e9060c6ad5092114fbba3c0becf000876f7 WatchSource:0}: Error finding container 9e2ac10b0dff16cf351ca51062109e9060c6ad5092114fbba3c0becf000876f7: Status 404 returned error can't find the container with id 9e2ac10b0dff16cf351ca51062109e9060c6ad5092114fbba3c0becf000876f7 Oct 30 18:24:33.537389 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.537331 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:33.537468 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.537415 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:33.537468 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.537456 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:33.541426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.541398 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n"] Oct 30 18:24:33.544254 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.544223 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:33.544320 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.544287 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:33.544353 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.544318 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:24:33.878892 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.878608 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" event={"ID":"352fefd4-5ce5-4ce2-81ff-28769154e70b","Type":"ContainerStarted","Data":"e9ed2e2dea302a6b56108124bc5cc3debc43fe6a6f0700f69de5c79b309b010d"} Oct 30 18:24:33.878892 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.878674 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" event={"ID":"352fefd4-5ce5-4ce2-81ff-28769154e70b","Type":"ContainerStarted","Data":"9e2ac10b0dff16cf351ca51062109e9060c6ad5092114fbba3c0becf000876f7"} Oct 30 18:24:33.880693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.880660 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" event={"ID":"bf05a858-37f7-4ce6-a4ba-5431a372d1b3","Type":"ContainerStarted","Data":"d80b9482c4286db5bd9b75ee0ce8bd8f826ff6b8d324580e6a2f7d12dcd678fb"} Oct 30 18:24:33.880853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.880718 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" event={"ID":"bf05a858-37f7-4ce6-a4ba-5431a372d1b3","Type":"ContainerStarted","Data":"61f585b92473d487e8c81e44cafbfa6225b80d4110cd0d2057416040ea47eb40"} Oct 30 18:24:33.882816 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.882793 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" event={"ID":"d1fd255d-3e8c-489a-9151-cb9533af2405","Type":"ContainerStarted","Data":"2c34e13eac894b19ed5ae92d3d22c57afd4637ac1255e4a25e2dccc5862ad943"} Oct 30 18:24:33.882918 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:33.882839 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" event={"ID":"d1fd255d-3e8c-489a-9151-cb9533af2405","Type":"ContainerStarted","Data":"4b1a97298b0a8027e4fcfe091980ed255740466aa32814ca7e7a0b043416c739"} Oct 30 18:24:34.014406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.014337 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" podStartSLOduration=2.014314538 podStartE2EDuration="2.014314538s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:33.923127321 +0000 UTC m=+1373.239547646" watchObservedRunningTime="2026-10-30 18:24:34.014314538 +0000 UTC m=+1373.330734861" Oct 30 18:24:34.105617 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.094809 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" podStartSLOduration=2.094784621 podStartE2EDuration="2.094784621s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:34.024083313 +0000 UTC m=+1373.340503634" watchObservedRunningTime="2026-10-30 18:24:34.094784621 +0000 UTC m=+1373.411204943" Oct 30 18:24:34.193637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.114202 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:34.193637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.114255 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:34.193637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.159700 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:34.193637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.164044 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:24:34.193637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.164095 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:24:34.193637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.164450 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 18:24:34.193637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.164541 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" podUID="d1fd255d-3e8c-489a-9151-cb9533af2405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 18:24:34.199667 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.199640 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:34.199996 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.199906 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 18:24:34.210893 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.205659 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:34.210893 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.210676 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:24:34.210893 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.210844 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" podUID="bf05a858-37f7-4ce6-a4ba-5431a372d1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 18:24:34.211222 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.210755 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" podUID="352fefd4-5ce5-4ce2-81ff-28769154e70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 18:24:34.216533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.214190 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" podStartSLOduration=2.214171988 podStartE2EDuration="2.214171988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:34.112223929 +0000 UTC m=+1373.428644263" watchObservedRunningTime="2026-10-30 18:24:34.214171988 +0000 UTC m=+1373.530592309" Oct 30 18:24:34.817745 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.817711 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d5ba31c-c8f3-4467-be05-fd146702503e" path="/var/lib/kubelet/pods/5d5ba31c-c8f3-4467-be05-fd146702503e/volumes" Oct 30 18:24:34.893044 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.892989 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7"] Oct 30 18:24:34.922293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:34.920295 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-9ljg7"] Oct 30 18:24:35.099244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:35.099156 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:35.099244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:35.099217 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:35.171325 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:35.170933 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:24:35.171325 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:35.171002 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:24:35.172838 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:35.172802 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 18:24:35.173065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:35.173038 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" podUID="d1fd255d-3e8c-489a-9151-cb9533af2405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 18:24:35.197495 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:35.197458 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 18:24:35.198619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:35.198497 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" podUID="352fefd4-5ce5-4ce2-81ff-28769154e70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 18:24:35.200329 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:35.199856 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:24:35.200329 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:35.199903 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" podUID="bf05a858-37f7-4ce6-a4ba-5431a372d1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 18:24:36.097771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:36.097739 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:36.097956 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:36.097794 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:36.158186 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:36.158154 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 18:24:36.158327 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:36.158202 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" podUID="d1fd255d-3e8c-489a-9151-cb9533af2405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 18:24:36.163620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:36.163591 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:24:36.163742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:36.163643 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:24:36.193759 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:36.193728 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 18:24:36.193900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:36.193775 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" podUID="352fefd4-5ce5-4ce2-81ff-28769154e70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 18:24:36.198249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:36.198228 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:24:36.198561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:36.198273 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" podUID="bf05a858-37f7-4ce6-a4ba-5431a372d1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 18:24:36.807116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:36.807074 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c" path="/var/lib/kubelet/pods/5d8e3953-b0b1-4570-89bb-c76e2f3f0a0c/volumes" Oct 30 18:24:37.103576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:37.103305 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:37.109085 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:37.109047 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:37.159057 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:37.159015 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 18:24:37.159253 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:37.159078 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" podUID="d1fd255d-3e8c-489a-9151-cb9533af2405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 18:24:37.163567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:37.163496 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" start-of-body= Oct 30 18:24:37.163726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:37.163592 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": dial tcp 10.133.0.175:15021: connect: connection refused" Oct 30 18:24:37.193556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:37.193503 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" start-of-body= Oct 30 18:24:37.193731 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:37.193572 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" podUID="352fefd4-5ce5-4ce2-81ff-28769154e70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": dial tcp 10.133.0.178:15021: connect: connection refused" Oct 30 18:24:37.199072 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:37.199047 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:24:37.199403 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:37.199089 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" podUID="bf05a858-37f7-4ce6-a4ba-5431a372d1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 18:24:38.097786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.097753 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" start-of-body= Oct 30 18:24:38.097949 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.097802 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": dial tcp 10.133.0.174:15021: connect: connection refused" Oct 30 18:24:38.158375 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.158338 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" start-of-body= Oct 30 18:24:38.158556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.158392 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" podUID="d1fd255d-3e8c-489a-9151-cb9533af2405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": dial tcp 10.133.0.177:15021: connect: connection refused" Oct 30 18:24:38.168608 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.168576 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:38.169691 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.169670 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:24:38.198117 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.198093 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:38.198264 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.198241 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" start-of-body= Oct 30 18:24:38.198344 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.198290 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" podUID="bf05a858-37f7-4ce6-a4ba-5431a372d1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": dial tcp 10.133.0.179:15021: connect: connection refused" Oct 30 18:24:38.198467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.198364 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:38.199146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.199130 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:24:38.619471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.619439 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88"] Oct 30 18:24:38.624926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.624887 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" Oct 30 18:24:38.654480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.654445 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88"] Oct 30 18:24:38.673220 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.673184 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b187a263-e999-46e7-a1d6-4a44595f6e2d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88\" (UID: \"b187a263-e999-46e7-a1d6-4a44595f6e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" Oct 30 18:24:38.673400 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.673252 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b187a263-e999-46e7-a1d6-4a44595f6e2d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88\" (UID: \"b187a263-e999-46e7-a1d6-4a44595f6e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" Oct 30 18:24:38.673400 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.673279 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b187a263-e999-46e7-a1d6-4a44595f6e2d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88\" (UID: \"b187a263-e999-46e7-a1d6-4a44595f6e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" Oct 30 18:24:38.673553 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.673405 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5rb8\" (UniqueName: \"kubernetes.io/projected/b187a263-e999-46e7-a1d6-4a44595f6e2d-kube-api-access-t5rb8\") pod \"maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88\" (UID: \"b187a263-e999-46e7-a1d6-4a44595f6e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" Oct 30 18:24:38.675464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.675437 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w4-c22995-serving-cert\"" Oct 30 18:24:38.675714 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.675693 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpqx\"" Oct 30 18:24:38.675793 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.675785 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:24:38.774662 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.774634 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5rb8\" (UniqueName: \"kubernetes.io/projected/b187a263-e999-46e7-a1d6-4a44595f6e2d-kube-api-access-t5rb8\") pod \"maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88\" (UID: \"b187a263-e999-46e7-a1d6-4a44595f6e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" Oct 30 18:24:38.774957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.774930 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b187a263-e999-46e7-a1d6-4a44595f6e2d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88\" (UID: \"b187a263-e999-46e7-a1d6-4a44595f6e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" Oct 30 18:24:38.775055 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.774984 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b187a263-e999-46e7-a1d6-4a44595f6e2d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88\" (UID: \"b187a263-e999-46e7-a1d6-4a44595f6e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" Oct 30 18:24:38.775055 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.775010 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b187a263-e999-46e7-a1d6-4a44595f6e2d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88\" (UID: \"b187a263-e999-46e7-a1d6-4a44595f6e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" Oct 30 18:24:38.775936 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.775912 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b187a263-e999-46e7-a1d6-4a44595f6e2d-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88\" (UID: \"b187a263-e999-46e7-a1d6-4a44595f6e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" Oct 30 18:24:38.778167 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.778100 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b187a263-e999-46e7-a1d6-4a44595f6e2d-metrics-certs\") pod \"maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88\" (UID: \"b187a263-e999-46e7-a1d6-4a44595f6e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" Oct 30 18:24:38.778167 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.778140 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b187a263-e999-46e7-a1d6-4a44595f6e2d-maas-api-tls\") pod \"maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88\" (UID: \"b187a263-e999-46e7-a1d6-4a44595f6e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" Oct 30 18:24:38.794455 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.794427 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5rb8\" (UniqueName: \"kubernetes.io/projected/b187a263-e999-46e7-a1d6-4a44595f6e2d-kube-api-access-t5rb8\") pod \"maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88\" (UID: \"b187a263-e999-46e7-a1d6-4a44595f6e2d\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" Oct 30 18:24:38.966128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:38.966054 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" Oct 30 18:24:39.505089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.465399 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:39.505089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.494369 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:39.505089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.494496 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:39.505089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.501872 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:39.534725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.525204 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:24:39.578303 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.578267 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm"] Oct 30 18:24:39.587706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.586264 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:39.587908 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.587888 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm" Oct 30 18:24:39.592992 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.588271 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:24:39.593245 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.593227 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:24:39.597991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.597958 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm"] Oct 30 18:24:39.607820 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.607787 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6"] Oct 30 18:24:39.612835 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.612813 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-c22995\"" Oct 30 18:24:39.613112 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.613097 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-c22995-dockercfg-k5257\"" Oct 30 18:24:39.627744 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.627700 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1053887b-8b36-407c-8427-af7207e989b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm\" (UID: \"1053887b-8b36-407c-8427-af7207e989b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm" Oct 30 18:24:39.631064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.631036 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8tzn\" (UniqueName: \"kubernetes.io/projected/1053887b-8b36-407c-8427-af7207e989b6-kube-api-access-c8tzn\") pod \"payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm\" (UID: \"1053887b-8b36-407c-8427-af7207e989b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm" Oct 30 18:24:39.632595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.631825 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6" Oct 30 18:24:39.712330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.651888 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6"] Oct 30 18:24:40.088535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.739357 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1053887b-8b36-407c-8427-af7207e989b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm\" (UID: \"1053887b-8b36-407c-8427-af7207e989b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm" Oct 30 18:24:40.096560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:40.096525 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8tzn\" (UniqueName: \"kubernetes.io/projected/1053887b-8b36-407c-8427-af7207e989b6-kube-api-access-c8tzn\") pod \"payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm\" (UID: \"1053887b-8b36-407c-8427-af7207e989b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm" Oct 30 18:24:40.129559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:39.872644 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1053887b-8b36-407c-8427-af7207e989b6-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm\" (UID: \"1053887b-8b36-407c-8427-af7207e989b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm" Oct 30 18:24:40.135989 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:40.135958 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8tzn\" (UniqueName: \"kubernetes.io/projected/1053887b-8b36-407c-8427-af7207e989b6-kube-api-access-c8tzn\") pod \"payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm\" (UID: \"1053887b-8b36-407c-8427-af7207e989b6\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm" Oct 30 18:24:40.232021 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:40.229014 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pt877\" (UniqueName: \"kubernetes.io/projected/7703d89f-2407-4aa1-b263-1de901a8162b-kube-api-access-pt877\") pod \"payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6\" (UID: \"7703d89f-2407-4aa1-b263-1de901a8162b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6" Oct 30 18:24:40.232021 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:40.229397 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7703d89f-2407-4aa1-b263-1de901a8162b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6\" (UID: \"7703d89f-2407-4aa1-b263-1de901a8162b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6" Oct 30 18:24:40.438268 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:40.403852 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm" Oct 30 18:24:40.450824 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:40.450786 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7703d89f-2407-4aa1-b263-1de901a8162b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6\" (UID: \"7703d89f-2407-4aa1-b263-1de901a8162b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6" Oct 30 18:24:40.464531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:40.451430 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pt877\" (UniqueName: \"kubernetes.io/projected/7703d89f-2407-4aa1-b263-1de901a8162b-kube-api-access-pt877\") pod \"payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6\" (UID: \"7703d89f-2407-4aa1-b263-1de901a8162b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6" Oct 30 18:24:40.534630 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:40.472688 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7703d89f-2407-4aa1-b263-1de901a8162b-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6\" (UID: \"7703d89f-2407-4aa1-b263-1de901a8162b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6" Oct 30 18:24:40.663230 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:40.662159 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pt877\" (UniqueName: \"kubernetes.io/projected/7703d89f-2407-4aa1-b263-1de901a8162b-kube-api-access-pt877\") pod \"payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6\" (UID: \"7703d89f-2407-4aa1-b263-1de901a8162b\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6" Oct 30 18:24:40.738103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:40.738066 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88"] Oct 30 18:24:40.743753 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:40.743559 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6" Oct 30 18:24:40.746217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:40.746192 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w"] Oct 30 18:24:40.746452 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:40.746414 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" podUID="c7a2a719-e154-4072-93f2-ed366344dbbc" containerName="istio-proxy" containerID="cri-o://4b840d76c940168f16b1d2d52fe061fd0f8eeba0102c0483c1df1ad8c95469d3" gracePeriod=30 Oct 30 18:24:41.160741 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:41.160709 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm"] Oct 30 18:24:41.162095 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:24:41.162065 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1053887b_8b36_407c_8427_af7207e989b6.slice/crio-cb610cd309e4b1cf39164ea26aef0d03e5de0a083ce4d8262c5b29fd392f4c2c WatchSource:0}: Error finding container cb610cd309e4b1cf39164ea26aef0d03e5de0a083ce4d8262c5b29fd392f4c2c: Status 404 returned error can't find the container with id cb610cd309e4b1cf39164ea26aef0d03e5de0a083ce4d8262c5b29fd392f4c2c Oct 30 18:24:41.209934 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:41.209898 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm" event={"ID":"1053887b-8b36-407c-8427-af7207e989b6","Type":"ContainerStarted","Data":"cb610cd309e4b1cf39164ea26aef0d03e5de0a083ce4d8262c5b29fd392f4c2c"} Oct 30 18:24:41.211312 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:41.211281 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" event={"ID":"b187a263-e999-46e7-a1d6-4a44595f6e2d","Type":"ContainerStarted","Data":"bcaee940241f66a64e8622fa24c5b5f3698212c8e855748ecbdcfb0d46e6b8ec"} Oct 30 18:24:41.247472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:41.247450 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6"] Oct 30 18:24:41.249954 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:24:41.249925 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7703d89f_2407_4aa1_b263_1de901a8162b.slice/crio-623a8c0063d5dd9d06477417de26c911513acc1fd7f685fd232487628fe65d96 WatchSource:0}: Error finding container 623a8c0063d5dd9d06477417de26c911513acc1fd7f685fd232487628fe65d96: Status 404 returned error can't find the container with id 623a8c0063d5dd9d06477417de26c911513acc1fd7f685fd232487628fe65d96 Oct 30 18:24:42.224003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:42.223960 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6" event={"ID":"7703d89f-2407-4aa1-b263-1de901a8162b","Type":"ContainerStarted","Data":"eca962e46774c2c90c0be38cec888918908821719aae4bd861a922a10ae23ba5"} Oct 30 18:24:42.224548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:42.224027 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6" event={"ID":"7703d89f-2407-4aa1-b263-1de901a8162b","Type":"ContainerStarted","Data":"623a8c0063d5dd9d06477417de26c911513acc1fd7f685fd232487628fe65d96"} Oct 30 18:24:42.224548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:42.224061 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6" Oct 30 18:24:42.226189 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:42.226156 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm" event={"ID":"1053887b-8b36-407c-8427-af7207e989b6","Type":"ContainerStarted","Data":"dcbf7007f6177639bc92b41a727528c37fa3db5e12d7bff1b7d07d55a874ffde"} Oct 30 18:24:42.226294 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:42.226229 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm" Oct 30 18:24:42.271282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:42.271228 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6" podStartSLOduration=3.271210728 podStartE2EDuration="3.271210728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:42.265986395 +0000 UTC m=+1381.582406714" watchObservedRunningTime="2026-10-30 18:24:42.271210728 +0000 UTC m=+1381.587631049" Oct 30 18:24:42.308629 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:42.308577 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm" podStartSLOduration=3.308563161 podStartE2EDuration="3.308563161s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:42.306143984 +0000 UTC m=+1381.622564316" watchObservedRunningTime="2026-10-30 18:24:42.308563161 +0000 UTC m=+1381.624983482" Oct 30 18:24:43.236392 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:43.236360 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" event={"ID":"b187a263-e999-46e7-a1d6-4a44595f6e2d","Type":"ContainerStarted","Data":"5cd3b035bd947ccec0c63d5f43bb7a88151f964760a8175d90adec1f1c63828b"} Oct 30 18:24:43.236852 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:43.236670 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" Oct 30 18:24:43.280119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:43.280076 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" podStartSLOduration=3.542681205 podStartE2EDuration="5.280061827s" totalImagesPullingTime="1.737380622s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:43.279794633 +0000 UTC m=+1382.596214954" watchObservedRunningTime="2026-10-30 18:24:43.280061827 +0000 UTC m=+1382.596482191" Oct 30 18:24:45.476972 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:45.476936 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw"] Oct 30 18:24:45.481772 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:45.481752 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw" Oct 30 18:24:45.501682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:45.501656 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw"] Oct 30 18:24:45.508864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:45.508821 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:24:45.519643 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:45.519613 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:24:45.550724 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:45.550685 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e212d8b5-6490-48e7-b2f4-636d017e9b29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw\" (UID: \"e212d8b5-6490-48e7-b2f4-636d017e9b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw" Oct 30 18:24:45.550949 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:45.550799 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zx2z\" (UniqueName: \"kubernetes.io/projected/e212d8b5-6490-48e7-b2f4-636d017e9b29-kube-api-access-8zx2z\") pod \"maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw\" (UID: \"e212d8b5-6490-48e7-b2f4-636d017e9b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw" Oct 30 18:24:45.551126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:45.551096 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e212d8b5-6490-48e7-b2f4-636d017e9b29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw\" (UID: \"e212d8b5-6490-48e7-b2f4-636d017e9b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw" Oct 30 18:24:45.652782 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:45.652750 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e212d8b5-6490-48e7-b2f4-636d017e9b29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw\" (UID: \"e212d8b5-6490-48e7-b2f4-636d017e9b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw" Oct 30 18:24:45.652975 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:45.652955 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e212d8b5-6490-48e7-b2f4-636d017e9b29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw\" (UID: \"e212d8b5-6490-48e7-b2f4-636d017e9b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw" Oct 30 18:24:45.653343 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:45.652999 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zx2z\" (UniqueName: \"kubernetes.io/projected/e212d8b5-6490-48e7-b2f4-636d017e9b29-kube-api-access-8zx2z\") pod \"maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw\" (UID: \"e212d8b5-6490-48e7-b2f4-636d017e9b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw" Oct 30 18:24:45.653721 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:45.653696 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e212d8b5-6490-48e7-b2f4-636d017e9b29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw\" (UID: \"e212d8b5-6490-48e7-b2f4-636d017e9b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw" Oct 30 18:24:45.672536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:45.656449 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e212d8b5-6490-48e7-b2f4-636d017e9b29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw\" (UID: \"e212d8b5-6490-48e7-b2f4-636d017e9b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw" Oct 30 18:24:45.728589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:45.695656 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zx2z\" (UniqueName: \"kubernetes.io/projected/e212d8b5-6490-48e7-b2f4-636d017e9b29-kube-api-access-8zx2z\") pod \"maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw\" (UID: \"e212d8b5-6490-48e7-b2f4-636d017e9b29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw" Oct 30 18:24:45.803534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:45.803088 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw" Oct 30 18:24:46.268259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.268229 5232 generic.go:356] "Generic (PLEG): container finished" podID="c7a2a719-e154-4072-93f2-ed366344dbbc" containerID="4b840d76c940168f16b1d2d52fe061fd0f8eeba0102c0483c1df1ad8c95469d3" exitCode=0 Oct 30 18:24:46.268452 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.268287 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" event={"ID":"c7a2a719-e154-4072-93f2-ed366344dbbc","Type":"ContainerDied","Data":"4b840d76c940168f16b1d2d52fe061fd0f8eeba0102c0483c1df1ad8c95469d3"} Oct 30 18:24:46.386158 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.386130 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:24:46.419438 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.419412 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-envoy\") pod \"c7a2a719-e154-4072-93f2-ed366344dbbc\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " Oct 30 18:24:46.419634 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.419462 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-workload-certs\") pod \"c7a2a719-e154-4072-93f2-ed366344dbbc\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " Oct 30 18:24:46.419634 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.419494 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-podinfo\") pod \"c7a2a719-e154-4072-93f2-ed366344dbbc\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " Oct 30 18:24:46.419634 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.419552 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-data\") pod \"c7a2a719-e154-4072-93f2-ed366344dbbc\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " Oct 30 18:24:46.419798 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.419677 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a2a719-e154-4072-93f2-ed366344dbbc-kube-api-access-jnzm7\" (UniqueName: \"kubernetes.io/projected/c7a2a719-e154-4072-93f2-ed366344dbbc-kube-api-access-jnzm7\") pod \"c7a2a719-e154-4072-93f2-ed366344dbbc\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " Oct 30 18:24:46.419798 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.419725 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-credential-socket\") pod \"c7a2a719-e154-4072-93f2-ed366344dbbc\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " Oct 30 18:24:46.419798 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.419765 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-workload-certs" pod "c7a2a719-e154-4072-93f2-ed366344dbbc" (UID: "c7a2a719-e154-4072-93f2-ed366344dbbc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.420031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.419851 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c7a2a719-e154-4072-93f2-ed366344dbbc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a2a719-e154-4072-93f2-ed366344dbbc-istiod-ca-cert\") pod \"c7a2a719-e154-4072-93f2-ed366344dbbc\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " Oct 30 18:24:46.420031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.419935 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-data" pod "c7a2a719-e154-4072-93f2-ed366344dbbc" (UID: "c7a2a719-e154-4072-93f2-ed366344dbbc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.420130 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.420067 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-credential-socket" pod "c7a2a719-e154-4072-93f2-ed366344dbbc" (UID: "c7a2a719-e154-4072-93f2-ed366344dbbc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.420130 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.420104 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-token\" (UniqueName: \"kubernetes.io/projected/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-token\") pod \"c7a2a719-e154-4072-93f2-ed366344dbbc\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " Oct 30 18:24:46.420237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.420146 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c7a2a719-e154-4072-93f2-ed366344dbbc-istiod-ca-cert" pod "c7a2a719-e154-4072-93f2-ed366344dbbc" (UID: "c7a2a719-e154-4072-93f2-ed366344dbbc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:46.420237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.420154 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-workload-socket\") pod \"c7a2a719-e154-4072-93f2-ed366344dbbc\" (UID: \"c7a2a719-e154-4072-93f2-ed366344dbbc\") " Oct 30 18:24:46.420464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.420421 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-workload-socket" pod "c7a2a719-e154-4072-93f2-ed366344dbbc" (UID: "c7a2a719-e154-4072-93f2-ed366344dbbc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.421280 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.421253 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.421397 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.421283 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.421397 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.421297 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.421397 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.421309 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.421397 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.421323 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c7a2a719-e154-4072-93f2-ed366344dbbc-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.422027 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.421990 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-podinfo" pod "c7a2a719-e154-4072-93f2-ed366344dbbc" (UID: "c7a2a719-e154-4072-93f2-ed366344dbbc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:24:46.422124 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.422091 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-envoy" pod "c7a2a719-e154-4072-93f2-ed366344dbbc" (UID: "c7a2a719-e154-4072-93f2-ed366344dbbc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:24:46.422250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.422230 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a2a719-e154-4072-93f2-ed366344dbbc-kube-api-access-jnzm7" pod "c7a2a719-e154-4072-93f2-ed366344dbbc" (UID: "c7a2a719-e154-4072-93f2-ed366344dbbc"). InnerVolumeSpecName "kube-api-access-jnzm7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:46.422477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.422458 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-token" pod "c7a2a719-e154-4072-93f2-ed366344dbbc" (UID: "c7a2a719-e154-4072-93f2-ed366344dbbc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:46.481131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.481105 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw"] Oct 30 18:24:46.484176 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:24:46.484138 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode212d8b5_6490_48e7_b2f4_636d017e9b29.slice/crio-d14ff1bf53b731dc47a2f0eb643aa04a1eb0866cedf1867dbcc6d5fa647a8e2f WatchSource:0}: Error finding container d14ff1bf53b731dc47a2f0eb643aa04a1eb0866cedf1867dbcc6d5fa647a8e2f: Status 404 returned error can't find the container with id d14ff1bf53b731dc47a2f0eb643aa04a1eb0866cedf1867dbcc6d5fa647a8e2f Oct 30 18:24:46.522695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.522672 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.522695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.522695 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.522835 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.522704 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c7a2a719-e154-4072-93f2-ed366344dbbc-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:46.522835 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:46.522713 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jnzm7\" (UniqueName: \"kubernetes.io/projected/c7a2a719-e154-4072-93f2-ed366344dbbc-kube-api-access-jnzm7\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:47.279032 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:47.278957 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" Oct 30 18:24:47.279032 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:47.278990 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w" event={"ID":"c7a2a719-e154-4072-93f2-ed366344dbbc","Type":"ContainerDied","Data":"2bf9ac057f40dec9a91a203c481943ee2e3ef24bf8c41c25e90721c59f6da833"} Oct 30 18:24:47.279278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:47.279086 5232 scope.go:122] "RemoveContainer" containerID="4b840d76c940168f16b1d2d52fe061fd0f8eeba0102c0483c1df1ad8c95469d3" Oct 30 18:24:47.281074 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:47.281047 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw" event={"ID":"e212d8b5-6490-48e7-b2f4-636d017e9b29","Type":"ContainerStarted","Data":"5276db729f7c9a48690ba54439a837dde86b28890a5b127d08ec28d76ee4db84"} Oct 30 18:24:47.281198 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:47.281096 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw" event={"ID":"e212d8b5-6490-48e7-b2f4-636d017e9b29","Type":"ContainerStarted","Data":"d14ff1bf53b731dc47a2f0eb643aa04a1eb0866cedf1867dbcc6d5fa647a8e2f"} Oct 30 18:24:47.378544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:47.377689 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw" podStartSLOduration=2.377663919 podStartE2EDuration="2.377663919s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:47.365958055 +0000 UTC m=+1386.682378378" watchObservedRunningTime="2026-10-30 18:24:47.377663919 +0000 UTC m=+1386.694084237" Oct 30 18:24:47.445586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:47.442718 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w"] Oct 30 18:24:47.448323 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:47.446986 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-7m67w"] Oct 30 18:24:47.806909 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:47.806873 5232 scope.go:122] "RemoveContainer" containerID="bc9a8cadf000d65946c2a97a655b30b0fc32927d9070aee539a27b3a2248058e" Oct 30 18:24:47.820101 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:47.820082 5232 scope.go:122] "RemoveContainer" containerID="a00d6fa99fc9d539c5bde35eece222551e33dcbbd2631163ad2f05fddd002fd8" Oct 30 18:24:47.830736 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:47.830711 5232 scope.go:122] "RemoveContainer" containerID="efc2a4ba046c5b48ebb669e05b540084ea4587784f1242ebbde70b22ae74dcf4" Oct 30 18:24:47.841166 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:47.841146 5232 scope.go:122] "RemoveContainer" containerID="c34fb0a5e29c945f8483f371d36a77529185e0cb36da2391615176a1c11afcd3" Oct 30 18:24:47.851157 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:47.851133 5232 scope.go:122] "RemoveContainer" containerID="b32b9216c48eb085ea7fee9803dd6a09566f0ae3962e99c3a4d5c59322966421" Oct 30 18:24:47.860380 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:47.860359 5232 scope.go:122] "RemoveContainer" containerID="5cef4ca50a5e450fc0cf4a4fb2f901fb6e8dc99c6897bfaee0842d673b94e425" Oct 30 18:24:47.869659 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:47.869640 5232 scope.go:122] "RemoveContainer" containerID="4183454e4799771070fcaa68537ac7ec2a27f0e7b8d16d5db074aaae84037aa6" Oct 30 18:24:47.878836 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:47.878817 5232 scope.go:122] "RemoveContainer" containerID="bbfb4e4052046f8e756ba31c4fd5dbadb05550a0589c14aa472536a6fae80a00" Oct 30 18:24:48.307584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:48.303168 5232 generic.go:356] "Generic (PLEG): container finished" podID="e212d8b5-6490-48e7-b2f4-636d017e9b29" containerID="5276db729f7c9a48690ba54439a837dde86b28890a5b127d08ec28d76ee4db84" exitCode=0 Oct 30 18:24:48.307584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:48.303351 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw" event={"ID":"e212d8b5-6490-48e7-b2f4-636d017e9b29","Type":"ContainerDied","Data":"5276db729f7c9a48690ba54439a837dde86b28890a5b127d08ec28d76ee4db84"} Oct 30 18:24:48.806153 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:48.806124 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c7a2a719-e154-4072-93f2-ed366344dbbc" path="/var/lib/kubelet/pods/c7a2a719-e154-4072-93f2-ed366344dbbc/volumes" Oct 30 18:24:49.251210 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:49.251185 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" Oct 30 18:24:49.521688 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:49.521624 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw" Oct 30 18:24:49.595257 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:49.595227 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e212d8b5-6490-48e7-b2f4-636d017e9b29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e212d8b5-6490-48e7-b2f4-636d017e9b29-maas-api-xks-ca\") pod \"e212d8b5-6490-48e7-b2f4-636d017e9b29\" (UID: \"e212d8b5-6490-48e7-b2f4-636d017e9b29\") " Oct 30 18:24:49.595439 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:49.595330 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e212d8b5-6490-48e7-b2f4-636d017e9b29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e212d8b5-6490-48e7-b2f4-636d017e9b29-maas-api-openshift-ca\") pod \"e212d8b5-6490-48e7-b2f4-636d017e9b29\" (UID: \"e212d8b5-6490-48e7-b2f4-636d017e9b29\") " Oct 30 18:24:49.595499 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:49.595438 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e212d8b5-6490-48e7-b2f4-636d017e9b29-kube-api-access-8zx2z\" (UniqueName: \"kubernetes.io/projected/e212d8b5-6490-48e7-b2f4-636d017e9b29-kube-api-access-8zx2z\") pod \"e212d8b5-6490-48e7-b2f4-636d017e9b29\" (UID: \"e212d8b5-6490-48e7-b2f4-636d017e9b29\") " Oct 30 18:24:49.595815 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:49.595786 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e212d8b5-6490-48e7-b2f4-636d017e9b29-maas-api-openshift-ca" pod "e212d8b5-6490-48e7-b2f4-636d017e9b29" (UID: "e212d8b5-6490-48e7-b2f4-636d017e9b29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:24:49.596858 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:49.596830 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e212d8b5-6490-48e7-b2f4-636d017e9b29-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:49.598573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:49.598490 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e212d8b5-6490-48e7-b2f4-636d017e9b29-kube-api-access-8zx2z" pod "e212d8b5-6490-48e7-b2f4-636d017e9b29" (UID: "e212d8b5-6490-48e7-b2f4-636d017e9b29"). InnerVolumeSpecName "kube-api-access-8zx2z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:24:49.598688 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:49.598614 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e212d8b5-6490-48e7-b2f4-636d017e9b29-maas-api-xks-ca" pod "e212d8b5-6490-48e7-b2f4-636d017e9b29" (UID: "e212d8b5-6490-48e7-b2f4-636d017e9b29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:24:49.699018 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:49.698984 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e212d8b5-6490-48e7-b2f4-636d017e9b29-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:49.699018 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:49.699015 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zx2z\" (UniqueName: \"kubernetes.io/projected/e212d8b5-6490-48e7-b2f4-636d017e9b29-kube-api-access-8zx2z\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:24:50.327326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:50.327299 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw" Oct 30 18:24:50.327735 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:50.327297 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw" event={"ID":"e212d8b5-6490-48e7-b2f4-636d017e9b29","Type":"ContainerDied","Data":"d14ff1bf53b731dc47a2f0eb643aa04a1eb0866cedf1867dbcc6d5fa647a8e2f"} Oct 30 18:24:50.327735 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:50.327424 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d14ff1bf53b731dc47a2f0eb643aa04a1eb0866cedf1867dbcc6d5fa647a8e2f" Oct 30 18:24:53.274533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:53.270696 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm" Oct 30 18:24:53.274533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:53.271935 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6" Oct 30 18:24:56.142790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:56.142751 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b"] Oct 30 18:24:56.216536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:56.211699 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9"] Oct 30 18:24:56.691904 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:56.691864 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx"] Oct 30 18:24:56.692173 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:56.692140 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" podUID="ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" containerName="istio-proxy" containerID="cri-o://3a47235c7e61b1fe8c5b3590996f2ce2e23a04972f2c8cd2d9821ae7f94b92f5" gracePeriod=30 Oct 30 18:24:56.699081 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:56.699057 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4"] Oct 30 18:24:56.699309 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:56.699265 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" podUID="8a806882-2d84-427c-8698-9ed54994b748" containerName="istio-proxy" containerID="cri-o://197ba80f6b5e2c19aac5628d61b3f598aefe816d37e1b324df018c5dacf4cd4b" gracePeriod=30 Oct 30 18:24:56.704859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:56.704838 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt"] Oct 30 18:24:56.705056 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:56.705021 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" podUID="7403bbb1-5c6e-431f-88a2-1005b4a22421" containerName="istio-proxy" containerID="cri-o://74774892dbfbb1968405fcfb47f5d0201b3a56715cedc289a60c85aeb68bb854" gracePeriod=30 Oct 30 18:24:56.709416 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:56.709387 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642"] Oct 30 18:24:56.709700 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:56.709627 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" podUID="f2ebd246-b899-4a1d-9067-3993742fb028" containerName="istio-proxy" containerID="cri-o://aced874e598d3925ad113b36279d79937cb7ebbc84ed299c2493a0ff8f63e602" gracePeriod=30 Oct 30 18:24:57.277098 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:24:57.277054 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:24:57.279829 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.279799 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9"] Oct 30 18:24:57.281163 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.281138 5232 cpu_manager.go:414] "containerMap: removing container" podUID="c7a2a719-e154-4072-93f2-ed366344dbbc" containerName="istio-proxy" Oct 30 18:24:57.281163 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.281164 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c7a2a719-e154-4072-93f2-ed366344dbbc" containerName="istio-proxy" Oct 30 18:24:57.281349 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.281259 5232 cpu_manager.go:414] "containerMap: removing container" podUID="e212d8b5-6490-48e7-b2f4-636d017e9b29" containerName="revoke-keys" Oct 30 18:24:57.281349 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.281268 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e212d8b5-6490-48e7-b2f4-636d017e9b29" containerName="revoke-keys" Oct 30 18:24:57.281484 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.281437 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="e212d8b5-6490-48e7-b2f4-636d017e9b29" containerName="revoke-keys" Oct 30 18:24:57.281484 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.281455 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="c7a2a719-e154-4072-93f2-ed366344dbbc" containerName="istio-proxy" Oct 30 18:24:57.288557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.288120 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d"] Oct 30 18:24:57.288557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.288235 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9" Oct 30 18:24:57.293736 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.293711 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d" Oct 30 18:24:57.314491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.314463 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9"] Oct 30 18:24:57.316048 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.316020 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d"] Oct 30 18:24:57.334809 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.334784 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:24:57.335099 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.335076 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:24:57.420822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.420786 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cf6fe24-jflc9\" (UID: \"82285c8d-b803-4f2d-9ca7-ef302cf7b5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9" Oct 30 18:24:57.420990 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.420954 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546beae5-c98b-4b49-9bc8-46b77e64648c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19c62263-q8f7d\" (UID: \"546beae5-c98b-4b49-9bc8-46b77e64648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d" Oct 30 18:24:57.421104 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.421084 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546beae5-c98b-4b49-9bc8-46b77e64648c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19c62263-q8f7d\" (UID: \"546beae5-c98b-4b49-9bc8-46b77e64648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d" Oct 30 18:24:57.421194 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.421180 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cf6fe24-jflc9\" (UID: \"82285c8d-b803-4f2d-9ca7-ef302cf7b5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9" Oct 30 18:24:57.421269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.421258 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s22h\" (UniqueName: \"kubernetes.io/projected/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-kube-api-access-7s22h\") pod \"maas-api-revoke-sub-2cf6fe24-jflc9\" (UID: \"82285c8d-b803-4f2d-9ca7-ef302cf7b5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9" Oct 30 18:24:57.421424 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.421404 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz76g\" (UniqueName: \"kubernetes.io/projected/546beae5-c98b-4b49-9bc8-46b77e64648c-kube-api-access-wz76g\") pod \"maas-api-revoke-sub-19c62263-q8f7d\" (UID: \"546beae5-c98b-4b49-9bc8-46b77e64648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d" Oct 30 18:24:57.523154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.523107 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s22h\" (UniqueName: \"kubernetes.io/projected/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-kube-api-access-7s22h\") pod \"maas-api-revoke-sub-2cf6fe24-jflc9\" (UID: \"82285c8d-b803-4f2d-9ca7-ef302cf7b5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9" Oct 30 18:24:57.523334 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.523203 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz76g\" (UniqueName: \"kubernetes.io/projected/546beae5-c98b-4b49-9bc8-46b77e64648c-kube-api-access-wz76g\") pod \"maas-api-revoke-sub-19c62263-q8f7d\" (UID: \"546beae5-c98b-4b49-9bc8-46b77e64648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d" Oct 30 18:24:57.523334 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.523261 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cf6fe24-jflc9\" (UID: \"82285c8d-b803-4f2d-9ca7-ef302cf7b5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9" Oct 30 18:24:57.523457 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.523344 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546beae5-c98b-4b49-9bc8-46b77e64648c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19c62263-q8f7d\" (UID: \"546beae5-c98b-4b49-9bc8-46b77e64648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d" Oct 30 18:24:57.523457 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.523423 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546beae5-c98b-4b49-9bc8-46b77e64648c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19c62263-q8f7d\" (UID: \"546beae5-c98b-4b49-9bc8-46b77e64648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d" Oct 30 18:24:57.523649 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.523628 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cf6fe24-jflc9\" (UID: \"82285c8d-b803-4f2d-9ca7-ef302cf7b5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9" Oct 30 18:24:57.524393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.524362 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2cf6fe24-jflc9\" (UID: \"82285c8d-b803-4f2d-9ca7-ef302cf7b5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9" Oct 30 18:24:57.524522 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.524367 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546beae5-c98b-4b49-9bc8-46b77e64648c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-19c62263-q8f7d\" (UID: \"546beae5-c98b-4b49-9bc8-46b77e64648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d" Oct 30 18:24:57.526340 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.526315 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546beae5-c98b-4b49-9bc8-46b77e64648c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-19c62263-q8f7d\" (UID: \"546beae5-c98b-4b49-9bc8-46b77e64648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d" Oct 30 18:24:57.526610 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.526590 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2cf6fe24-jflc9\" (UID: \"82285c8d-b803-4f2d-9ca7-ef302cf7b5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9" Oct 30 18:24:57.545043 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.544970 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz76g\" (UniqueName: \"kubernetes.io/projected/546beae5-c98b-4b49-9bc8-46b77e64648c-kube-api-access-wz76g\") pod \"maas-api-revoke-sub-19c62263-q8f7d\" (UID: \"546beae5-c98b-4b49-9bc8-46b77e64648c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d" Oct 30 18:24:57.546909 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.546888 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s22h\" (UniqueName: \"kubernetes.io/projected/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-kube-api-access-7s22h\") pod \"maas-api-revoke-sub-2cf6fe24-jflc9\" (UID: \"82285c8d-b803-4f2d-9ca7-ef302cf7b5fc\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9" Oct 30 18:24:57.633817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.632799 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9" Oct 30 18:24:57.643544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:57.640786 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d" Oct 30 18:24:58.166316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:58.166288 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d"] Oct 30 18:24:58.166464 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:24:58.166442 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod546beae5_c98b_4b49_9bc8_46b77e64648c.slice/crio-e0e1f52cca261660cfa21acdaa03194bf71490380162846801e8266080ef34d2 WatchSource:0}: Error finding container e0e1f52cca261660cfa21acdaa03194bf71490380162846801e8266080ef34d2: Status 404 returned error can't find the container with id e0e1f52cca261660cfa21acdaa03194bf71490380162846801e8266080ef34d2 Oct 30 18:24:58.183194 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:24:58.183161 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82285c8d_b803_4f2d_9ca7_ef302cf7b5fc.slice/crio-895bc07794a863ec4dcbdcd138f07bf57df54007ac67123f413f37ea688502b7 WatchSource:0}: Error finding container 895bc07794a863ec4dcbdcd138f07bf57df54007ac67123f413f37ea688502b7: Status 404 returned error can't find the container with id 895bc07794a863ec4dcbdcd138f07bf57df54007ac67123f413f37ea688502b7 Oct 30 18:24:58.183802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:58.183778 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9"] Oct 30 18:24:58.722805 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:58.722773 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9" event={"ID":"82285c8d-b803-4f2d-9ca7-ef302cf7b5fc","Type":"ContainerStarted","Data":"895bc07794a863ec4dcbdcd138f07bf57df54007ac67123f413f37ea688502b7"} Oct 30 18:24:58.724802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:58.724773 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d" event={"ID":"546beae5-c98b-4b49-9bc8-46b77e64648c","Type":"ContainerStarted","Data":"790a2963ad1401ef42558e95474b87268990b93e611c14c11320ab2845b39370"} Oct 30 18:24:58.724924 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:58.724828 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d" event={"ID":"546beae5-c98b-4b49-9bc8-46b77e64648c","Type":"ContainerStarted","Data":"e0e1f52cca261660cfa21acdaa03194bf71490380162846801e8266080ef34d2"} Oct 30 18:24:58.763982 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:58.763929 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d" podStartSLOduration=1.763913755 podStartE2EDuration="1.763913755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:24:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:24:58.763551731 +0000 UTC m=+1398.079972054" watchObservedRunningTime="2026-10-30 18:24:58.763913755 +0000 UTC m=+1398.080334075" Oct 30 18:24:59.031811 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:59.031777 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59"] Oct 30 18:24:59.036414 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:59.036386 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0093b00c-2xj59"] Oct 30 18:24:59.747529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:59.747471 5232 generic.go:356] "Generic (PLEG): container finished" podID="546beae5-c98b-4b49-9bc8-46b77e64648c" containerID="790a2963ad1401ef42558e95474b87268990b93e611c14c11320ab2845b39370" exitCode=0 Oct 30 18:24:59.748182 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:59.748155 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d" event={"ID":"546beae5-c98b-4b49-9bc8-46b77e64648c","Type":"ContainerDied","Data":"790a2963ad1401ef42558e95474b87268990b93e611c14c11320ab2845b39370"} Oct 30 18:24:59.751392 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:59.751368 5232 generic.go:356] "Generic (PLEG): container finished" podID="82285c8d-b803-4f2d-9ca7-ef302cf7b5fc" containerID="5900e78954006843ceacdb34862b0d57d856c9772b93fe0a3414497a5db57085" exitCode=0 Oct 30 18:24:59.751900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:24:59.751654 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9" event={"ID":"82285c8d-b803-4f2d-9ca7-ef302cf7b5fc","Type":"ContainerDied","Data":"5900e78954006843ceacdb34862b0d57d856c9772b93fe0a3414497a5db57085"} Oct 30 18:25:00.813531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:00.810662 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c2d20871-227a-4ba9-82a7-ae48efbe014b" path="/var/lib/kubelet/pods/c2d20871-227a-4ba9-82a7-ae48efbe014b/volumes" Oct 30 18:25:00.949010 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:00.948978 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d" Oct 30 18:25:00.994203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:00.994179 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9" Oct 30 18:25:01.026561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.026444 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/546beae5-c98b-4b49-9bc8-46b77e64648c-kube-api-access-wz76g\" (UniqueName: \"kubernetes.io/projected/546beae5-c98b-4b49-9bc8-46b77e64648c-kube-api-access-wz76g\") pod \"546beae5-c98b-4b49-9bc8-46b77e64648c\" (UID: \"546beae5-c98b-4b49-9bc8-46b77e64648c\") " Oct 30 18:25:01.026710 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.026575 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/546beae5-c98b-4b49-9bc8-46b77e64648c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546beae5-c98b-4b49-9bc8-46b77e64648c-maas-api-xks-ca\") pod \"546beae5-c98b-4b49-9bc8-46b77e64648c\" (UID: \"546beae5-c98b-4b49-9bc8-46b77e64648c\") " Oct 30 18:25:01.026890 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.026870 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/546beae5-c98b-4b49-9bc8-46b77e64648c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546beae5-c98b-4b49-9bc8-46b77e64648c-maas-api-openshift-ca\") pod \"546beae5-c98b-4b49-9bc8-46b77e64648c\" (UID: \"546beae5-c98b-4b49-9bc8-46b77e64648c\") " Oct 30 18:25:01.027550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.027485 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/546beae5-c98b-4b49-9bc8-46b77e64648c-maas-api-openshift-ca" pod "546beae5-c98b-4b49-9bc8-46b77e64648c" (UID: "546beae5-c98b-4b49-9bc8-46b77e64648c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:01.028948 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.028915 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/546beae5-c98b-4b49-9bc8-46b77e64648c-kube-api-access-wz76g" pod "546beae5-c98b-4b49-9bc8-46b77e64648c" (UID: "546beae5-c98b-4b49-9bc8-46b77e64648c"). InnerVolumeSpecName "kube-api-access-wz76g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:01.029626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.029605 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/546beae5-c98b-4b49-9bc8-46b77e64648c-maas-api-xks-ca" pod "546beae5-c98b-4b49-9bc8-46b77e64648c" (UID: "546beae5-c98b-4b49-9bc8-46b77e64648c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:01.128494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.128461 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-maas-api-openshift-ca\") pod \"82285c8d-b803-4f2d-9ca7-ef302cf7b5fc\" (UID: \"82285c8d-b803-4f2d-9ca7-ef302cf7b5fc\") " Oct 30 18:25:01.128494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.128500 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-kube-api-access-7s22h\" (UniqueName: \"kubernetes.io/projected/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-kube-api-access-7s22h\") pod \"82285c8d-b803-4f2d-9ca7-ef302cf7b5fc\" (UID: \"82285c8d-b803-4f2d-9ca7-ef302cf7b5fc\") " Oct 30 18:25:01.128750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.128658 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-maas-api-xks-ca\") pod \"82285c8d-b803-4f2d-9ca7-ef302cf7b5fc\" (UID: \"82285c8d-b803-4f2d-9ca7-ef302cf7b5fc\") " Oct 30 18:25:01.128901 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.128876 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-maas-api-openshift-ca" pod "82285c8d-b803-4f2d-9ca7-ef302cf7b5fc" (UID: "82285c8d-b803-4f2d-9ca7-ef302cf7b5fc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:01.129251 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.129223 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz76g\" (UniqueName: \"kubernetes.io/projected/546beae5-c98b-4b49-9bc8-46b77e64648c-kube-api-access-wz76g\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:01.129251 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.129249 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/546beae5-c98b-4b49-9bc8-46b77e64648c-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:01.129403 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.129265 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:01.129403 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.129279 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/546beae5-c98b-4b49-9bc8-46b77e64648c-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:01.130889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.130865 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-maas-api-xks-ca" pod "82285c8d-b803-4f2d-9ca7-ef302cf7b5fc" (UID: "82285c8d-b803-4f2d-9ca7-ef302cf7b5fc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:01.130969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.130932 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-kube-api-access-7s22h" pod "82285c8d-b803-4f2d-9ca7-ef302cf7b5fc" (UID: "82285c8d-b803-4f2d-9ca7-ef302cf7b5fc"). InnerVolumeSpecName "kube-api-access-7s22h". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:01.162118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.162083 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" podUID="158c3671-9513-4f20-bdca-2801ec139c14" containerName="payload-processing" containerID="cri-o://1bb119c347160d9a63c813acb5d09ec9ff19ac59fb9624e8779d42973c717d44" gracePeriod=25 Oct 30 18:25:01.212486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.212429 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" podUID="964b7eb3-281f-4006-817e-6cb694f24dac" containerName="payload-pre-processing" containerID="cri-o://6f6ac0aa1ce4aed817bc78764f91776383b136572ea5987028f524f6f324c2d4" gracePeriod=25 Oct 30 18:25:01.231129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.231082 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:01.231129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.231133 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s22h\" (UniqueName: \"kubernetes.io/projected/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc-kube-api-access-7s22h\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:01.770031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.769993 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d" event={"ID":"546beae5-c98b-4b49-9bc8-46b77e64648c","Type":"ContainerDied","Data":"e0e1f52cca261660cfa21acdaa03194bf71490380162846801e8266080ef34d2"} Oct 30 18:25:01.770178 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.770059 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e0e1f52cca261660cfa21acdaa03194bf71490380162846801e8266080ef34d2" Oct 30 18:25:01.771620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.771600 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d" Oct 30 18:25:01.772073 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.772047 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9" event={"ID":"82285c8d-b803-4f2d-9ca7-ef302cf7b5fc","Type":"ContainerDied","Data":"895bc07794a863ec4dcbdcd138f07bf57df54007ac67123f413f37ea688502b7"} Oct 30 18:25:01.772170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.772095 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="895bc07794a863ec4dcbdcd138f07bf57df54007ac67123f413f37ea688502b7" Oct 30 18:25:01.772374 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:01.772355 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9" Oct 30 18:25:02.554973 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.554942 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:25:02.565556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.563135 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:25:02.566831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.566810 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:25:02.589300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.589267 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f2ebd246-b899-4a1d-9067-3993742fb028-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ebd246-b899-4a1d-9067-3993742fb028-istio-podinfo\") pod \"f2ebd246-b899-4a1d-9067-3993742fb028\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " Oct 30 18:25:02.589461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.589322 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-envoy\") pod \"7403bbb1-5c6e-431f-88a2-1005b4a22421\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " Oct 30 18:25:02.589461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.589372 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-podinfo\") pod \"7403bbb1-5c6e-431f-88a2-1005b4a22421\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " Oct 30 18:25:02.589461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.589453 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-istio-envoy\") pod \"8a806882-2d84-427c-8698-9ed54994b748\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " Oct 30 18:25:02.589681 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.589531 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ebd246-b899-4a1d-9067-3993742fb028-istio-token\" (UniqueName: \"kubernetes.io/projected/f2ebd246-b899-4a1d-9067-3993742fb028-istio-token\") pod \"f2ebd246-b899-4a1d-9067-3993742fb028\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " Oct 30 18:25:02.589681 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.589567 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a806882-2d84-427c-8698-9ed54994b748-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a806882-2d84-427c-8698-9ed54994b748-istio-podinfo\") pod \"8a806882-2d84-427c-8698-9ed54994b748\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " Oct 30 18:25:02.589681 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.589622 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a806882-2d84-427c-8698-9ed54994b748-istio-token\" (UniqueName: \"kubernetes.io/projected/8a806882-2d84-427c-8698-9ed54994b748-istio-token\") pod \"8a806882-2d84-427c-8698-9ed54994b748\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " Oct 30 18:25:02.589681 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.589672 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-workload-socket\") pod \"f2ebd246-b899-4a1d-9067-3993742fb028\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " Oct 30 18:25:02.589959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.589707 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-credential-socket\") pod \"7403bbb1-5c6e-431f-88a2-1005b4a22421\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " Oct 30 18:25:02.589959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.589751 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-workload-socket\") pod \"7403bbb1-5c6e-431f-88a2-1005b4a22421\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " Oct 30 18:25:02.589959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.589835 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-credential-socket\") pod \"f2ebd246-b899-4a1d-9067-3993742fb028\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " Oct 30 18:25:02.589959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.589868 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-workload-certs\") pod \"f2ebd246-b899-4a1d-9067-3993742fb028\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " Oct 30 18:25:02.589959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.589925 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-token\" (UniqueName: \"kubernetes.io/projected/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-token\") pod \"7403bbb1-5c6e-431f-88a2-1005b4a22421\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " Oct 30 18:25:02.590216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590021 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-workload-socket" pod "f2ebd246-b899-4a1d-9067-3993742fb028" (UID: "f2ebd246-b899-4a1d-9067-3993742fb028"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.590216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590048 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-istio-envoy\") pod \"f2ebd246-b899-4a1d-9067-3993742fb028\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " Oct 30 18:25:02.590216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590087 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-workload-certs\") pod \"7403bbb1-5c6e-431f-88a2-1005b4a22421\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " Oct 30 18:25:02.590216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590115 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7403bbb1-5c6e-431f-88a2-1005b4a22421-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7403bbb1-5c6e-431f-88a2-1005b4a22421-istiod-ca-cert\") pod \"7403bbb1-5c6e-431f-88a2-1005b4a22421\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " Oct 30 18:25:02.590216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590155 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-credential-socket\") pod \"8a806882-2d84-427c-8698-9ed54994b748\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " Oct 30 18:25:02.590467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590229 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f2ebd246-b899-4a1d-9067-3993742fb028-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebd246-b899-4a1d-9067-3993742fb028-istiod-ca-cert\") pod \"f2ebd246-b899-4a1d-9067-3993742fb028\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " Oct 30 18:25:02.590467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590255 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f2ebd246-b899-4a1d-9067-3993742fb028-kube-api-access-gcxg7\" (UniqueName: \"kubernetes.io/projected/f2ebd246-b899-4a1d-9067-3993742fb028-kube-api-access-gcxg7\") pod \"f2ebd246-b899-4a1d-9067-3993742fb028\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " Oct 30 18:25:02.590467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590310 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-workload-socket\") pod \"8a806882-2d84-427c-8698-9ed54994b748\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " Oct 30 18:25:02.590467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590336 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-istio-data\") pod \"f2ebd246-b899-4a1d-9067-3993742fb028\" (UID: \"f2ebd246-b899-4a1d-9067-3993742fb028\") " Oct 30 18:25:02.590467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590425 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-workload-certs\") pod \"8a806882-2d84-427c-8698-9ed54994b748\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " Oct 30 18:25:02.590762 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590484 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-credential-socket" pod "7403bbb1-5c6e-431f-88a2-1005b4a22421" (UID: "7403bbb1-5c6e-431f-88a2-1005b4a22421"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.590762 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590492 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-data\") pod \"7403bbb1-5c6e-431f-88a2-1005b4a22421\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " Oct 30 18:25:02.590762 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590619 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-istio-data\") pod \"8a806882-2d84-427c-8698-9ed54994b748\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " Oct 30 18:25:02.590762 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590670 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7403bbb1-5c6e-431f-88a2-1005b4a22421-kube-api-access-7wgpb\" (UniqueName: \"kubernetes.io/projected/7403bbb1-5c6e-431f-88a2-1005b4a22421-kube-api-access-7wgpb\") pod \"7403bbb1-5c6e-431f-88a2-1005b4a22421\" (UID: \"7403bbb1-5c6e-431f-88a2-1005b4a22421\") " Oct 30 18:25:02.590762 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590737 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a806882-2d84-427c-8698-9ed54994b748-kube-api-access-r66wj\" (UniqueName: \"kubernetes.io/projected/8a806882-2d84-427c-8698-9ed54994b748-kube-api-access-r66wj\") pod \"8a806882-2d84-427c-8698-9ed54994b748\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " Oct 30 18:25:02.590762 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590743 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-data" pod "7403bbb1-5c6e-431f-88a2-1005b4a22421" (UID: "7403bbb1-5c6e-431f-88a2-1005b4a22421"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.591079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590775 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a806882-2d84-427c-8698-9ed54994b748-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a806882-2d84-427c-8698-9ed54994b748-istiod-ca-cert\") pod \"8a806882-2d84-427c-8698-9ed54994b748\" (UID: \"8a806882-2d84-427c-8698-9ed54994b748\") " Oct 30 18:25:02.591079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590931 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-workload-certs" pod "7403bbb1-5c6e-431f-88a2-1005b4a22421" (UID: "7403bbb1-5c6e-431f-88a2-1005b4a22421"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.591079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.590947 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-workload-socket" pod "7403bbb1-5c6e-431f-88a2-1005b4a22421" (UID: "7403bbb1-5c6e-431f-88a2-1005b4a22421"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.591233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.591195 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7403bbb1-5c6e-431f-88a2-1005b4a22421-istiod-ca-cert" pod "7403bbb1-5c6e-431f-88a2-1005b4a22421" (UID: "7403bbb1-5c6e-431f-88a2-1005b4a22421"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:02.591740 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.591709 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a806882-2d84-427c-8698-9ed54994b748-istiod-ca-cert" pod "8a806882-2d84-427c-8698-9ed54994b748" (UID: "8a806882-2d84-427c-8698-9ed54994b748"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:02.591952 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.591929 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.592044 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.591958 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7403bbb1-5c6e-431f-88a2-1005b4a22421-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.592044 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.591971 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.592044 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.591985 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a806882-2d84-427c-8698-9ed54994b748-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.592044 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.591997 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.592044 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.592010 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.592044 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.592023 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.592044 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.591926 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-credential-socket" pod "8a806882-2d84-427c-8698-9ed54994b748" (UID: "8a806882-2d84-427c-8698-9ed54994b748"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.592044 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.591983 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-istio-data" pod "8a806882-2d84-427c-8698-9ed54994b748" (UID: "8a806882-2d84-427c-8698-9ed54994b748"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.592477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.592078 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2ebd246-b899-4a1d-9067-3993742fb028-istiod-ca-cert" pod "f2ebd246-b899-4a1d-9067-3993742fb028" (UID: "f2ebd246-b899-4a1d-9067-3993742fb028"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:02.592477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.592250 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-workload-socket" pod "8a806882-2d84-427c-8698-9ed54994b748" (UID: "8a806882-2d84-427c-8698-9ed54994b748"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.592477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.592403 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-workload-certs" pod "8a806882-2d84-427c-8698-9ed54994b748" (UID: "8a806882-2d84-427c-8698-9ed54994b748"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.592477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.592423 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-istio-data" pod "f2ebd246-b899-4a1d-9067-3993742fb028" (UID: "f2ebd246-b899-4a1d-9067-3993742fb028"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.592744 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.592553 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-workload-certs" pod "f2ebd246-b899-4a1d-9067-3993742fb028" (UID: "f2ebd246-b899-4a1d-9067-3993742fb028"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.592744 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.592643 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-credential-socket" pod "f2ebd246-b899-4a1d-9067-3993742fb028" (UID: "f2ebd246-b899-4a1d-9067-3993742fb028"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.595102 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.595053 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-istio-envoy" pod "8a806882-2d84-427c-8698-9ed54994b748" (UID: "8a806882-2d84-427c-8698-9ed54994b748"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.595393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.595332 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-istio-envoy" pod "f2ebd246-b899-4a1d-9067-3993742fb028" (UID: "f2ebd246-b899-4a1d-9067-3993742fb028"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.595748 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.595708 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a806882-2d84-427c-8698-9ed54994b748-istio-token" pod "8a806882-2d84-427c-8698-9ed54994b748" (UID: "8a806882-2d84-427c-8698-9ed54994b748"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:02.595950 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.595917 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a806882-2d84-427c-8698-9ed54994b748-istio-podinfo" pod "8a806882-2d84-427c-8698-9ed54994b748" (UID: "8a806882-2d84-427c-8698-9ed54994b748"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:02.596796 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.596774 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-token" pod "7403bbb1-5c6e-431f-88a2-1005b4a22421" (UID: "7403bbb1-5c6e-431f-88a2-1005b4a22421"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:02.597103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.597075 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ebd246-b899-4a1d-9067-3993742fb028-kube-api-access-gcxg7" pod "f2ebd246-b899-4a1d-9067-3993742fb028" (UID: "f2ebd246-b899-4a1d-9067-3993742fb028"). InnerVolumeSpecName "kube-api-access-gcxg7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:02.597913 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.597853 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f2ebd246-b899-4a1d-9067-3993742fb028-istio-podinfo" pod "f2ebd246-b899-4a1d-9067-3993742fb028" (UID: "f2ebd246-b899-4a1d-9067-3993742fb028"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:02.598110 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.598086 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7403bbb1-5c6e-431f-88a2-1005b4a22421-kube-api-access-7wgpb" pod "7403bbb1-5c6e-431f-88a2-1005b4a22421" (UID: "7403bbb1-5c6e-431f-88a2-1005b4a22421"). InnerVolumeSpecName "kube-api-access-7wgpb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:02.598252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.598227 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2ebd246-b899-4a1d-9067-3993742fb028-istio-token" pod "f2ebd246-b899-4a1d-9067-3993742fb028" (UID: "f2ebd246-b899-4a1d-9067-3993742fb028"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:02.598376 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.598305 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-podinfo" pod "7403bbb1-5c6e-431f-88a2-1005b4a22421" (UID: "7403bbb1-5c6e-431f-88a2-1005b4a22421"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:02.599272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.599249 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-envoy" pod "7403bbb1-5c6e-431f-88a2-1005b4a22421" (UID: "7403bbb1-5c6e-431f-88a2-1005b4a22421"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.599647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.599626 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a806882-2d84-427c-8698-9ed54994b748-kube-api-access-r66wj" pod "8a806882-2d84-427c-8698-9ed54994b748" (UID: "8a806882-2d84-427c-8698-9ed54994b748"). InnerVolumeSpecName "kube-api-access-r66wj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:02.648131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.648102 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:25:02.693048 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.693013 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-data\") pod \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " Oct 30 18:25:02.693228 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.693072 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istiod-ca-cert\") pod \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " Oct 30 18:25:02.693228 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.693159 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-credential-socket\") pod \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " Oct 30 18:25:02.693228 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.693186 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-token\" (UniqueName: \"kubernetes.io/projected/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-token\") pod \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " Oct 30 18:25:02.693406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.693294 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-envoy\") pod \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " Oct 30 18:25:02.693406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.693357 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-data" pod "ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" (UID: "ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.693557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.693447 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-podinfo\") pod \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " Oct 30 18:25:02.693557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.693498 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-workload-socket\") pod \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " Oct 30 18:25:02.693682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.693565 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istiod-ca-cert" pod "ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" (UID: "ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:02.693682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.693661 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-workload-certs\") pod \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " Oct 30 18:25:02.693793 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.693744 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-kube-api-access-ntbss\" (UniqueName: \"kubernetes.io/projected/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-kube-api-access-ntbss\") pod \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\" (UID: \"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e\") " Oct 30 18:25:02.693793 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.693754 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-credential-socket" pod "ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" (UID: "ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.693894 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.693855 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-workload-socket" pod "ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" (UID: "ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.694078 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.694048 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-workload-certs" pod "ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" (UID: "ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.694931 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.694904 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.694935 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.694948 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.694959 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.694971 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.694986 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f2ebd246-b899-4a1d-9067-3993742fb028-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695002 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gcxg7\" (UniqueName: \"kubernetes.io/projected/f2ebd246-b899-4a1d-9067-3993742fb028-kube-api-access-gcxg7\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695014 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695027 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695038 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f2ebd246-b899-4a1d-9067-3993742fb028-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695051 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695062 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695075 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695089 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695102 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7wgpb\" (UniqueName: \"kubernetes.io/projected/7403bbb1-5c6e-431f-88a2-1005b4a22421-kube-api-access-7wgpb\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695117 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r66wj\" (UniqueName: \"kubernetes.io/projected/8a806882-2d84-427c-8698-9ed54994b748-kube-api-access-r66wj\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695133 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f2ebd246-b899-4a1d-9067-3993742fb028-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695148 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695162 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695177 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7403bbb1-5c6e-431f-88a2-1005b4a22421-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695190 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a806882-2d84-427c-8698-9ed54994b748-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695202 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f2ebd246-b899-4a1d-9067-3993742fb028-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695214 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a806882-2d84-427c-8698-9ed54994b748-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695227 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.695377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.695242 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a806882-2d84-427c-8698-9ed54994b748-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.696075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.696053 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-token" pod "ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" (UID: "ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:02.696362 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.696335 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-envoy" pod "ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" (UID: "ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:02.696463 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.696426 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-podinfo" pod "ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" (UID: "ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:02.697089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.697067 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-kube-api-access-ntbss" pod "ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" (UID: "ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e"). InnerVolumeSpecName "kube-api-access-ntbss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:02.787056 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.787013 5232 generic.go:356] "Generic (PLEG): container finished" podID="7403bbb1-5c6e-431f-88a2-1005b4a22421" containerID="74774892dbfbb1968405fcfb47f5d0201b3a56715cedc289a60c85aeb68bb854" exitCode=0 Oct 30 18:25:02.787251 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.787103 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" Oct 30 18:25:02.787251 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.787182 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" event={"ID":"7403bbb1-5c6e-431f-88a2-1005b4a22421","Type":"ContainerDied","Data":"74774892dbfbb1968405fcfb47f5d0201b3a56715cedc289a60c85aeb68bb854"} Oct 30 18:25:02.787350 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.787251 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt" event={"ID":"7403bbb1-5c6e-431f-88a2-1005b4a22421","Type":"ContainerDied","Data":"7d3fcc8b33b08512b456b78b237ccd0e673109ebb855f7a556f62fc9e077d120"} Oct 30 18:25:02.787350 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.787284 5232 scope.go:122] "RemoveContainer" containerID="74774892dbfbb1968405fcfb47f5d0201b3a56715cedc289a60c85aeb68bb854" Oct 30 18:25:02.794316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.794278 5232 generic.go:356] "Generic (PLEG): container finished" podID="8a806882-2d84-427c-8698-9ed54994b748" containerID="197ba80f6b5e2c19aac5628d61b3f598aefe816d37e1b324df018c5dacf4cd4b" exitCode=0 Oct 30 18:25:02.794465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.794449 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" event={"ID":"8a806882-2d84-427c-8698-9ed54994b748","Type":"ContainerDied","Data":"197ba80f6b5e2c19aac5628d61b3f598aefe816d37e1b324df018c5dacf4cd4b"} Oct 30 18:25:02.794584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.794495 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" Oct 30 18:25:02.794886 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.794544 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" event={"ID":"8a806882-2d84-427c-8698-9ed54994b748","Type":"ContainerDied","Data":"9103bf0638175aaf6b18f0d7525b2fe6df92c53460b04a5974d30bee0bc1b197"} Oct 30 18:25:02.798490 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.798461 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntbss\" (UniqueName: \"kubernetes.io/projected/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-kube-api-access-ntbss\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.798625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.798496 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.798625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.798534 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.798625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.798547 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:02.798946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.798922 5232 generic.go:356] "Generic (PLEG): container finished" podID="ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" containerID="3a47235c7e61b1fe8c5b3590996f2ce2e23a04972f2c8cd2d9821ae7f94b92f5" exitCode=0 Oct 30 18:25:02.799094 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.799076 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" Oct 30 18:25:02.804451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.802921 5232 generic.go:356] "Generic (PLEG): container finished" podID="f2ebd246-b899-4a1d-9067-3993742fb028" containerID="aced874e598d3925ad113b36279d79937cb7ebbc84ed299c2493a0ff8f63e602" exitCode=0 Oct 30 18:25:02.804451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.803122 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" Oct 30 18:25:02.804451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.803650 5232 scope.go:122] "RemoveContainer" containerID="74774892dbfbb1968405fcfb47f5d0201b3a56715cedc289a60c85aeb68bb854" Oct 30 18:25:02.805849 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:25:02.804589 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"74774892dbfbb1968405fcfb47f5d0201b3a56715cedc289a60c85aeb68bb854\": container with ID starting with 74774892dbfbb1968405fcfb47f5d0201b3a56715cedc289a60c85aeb68bb854 not found: ID does not exist" containerID="74774892dbfbb1968405fcfb47f5d0201b3a56715cedc289a60c85aeb68bb854" Oct 30 18:25:02.805849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.804626 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"74774892dbfbb1968405fcfb47f5d0201b3a56715cedc289a60c85aeb68bb854"} err="failed to get container status \"74774892dbfbb1968405fcfb47f5d0201b3a56715cedc289a60c85aeb68bb854\": rpc error: code = NotFound desc = could not find container \"74774892dbfbb1968405fcfb47f5d0201b3a56715cedc289a60c85aeb68bb854\": container with ID starting with 74774892dbfbb1968405fcfb47f5d0201b3a56715cedc289a60c85aeb68bb854 not found: ID does not exist" Oct 30 18:25:02.805849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.804651 5232 scope.go:122] "RemoveContainer" containerID="197ba80f6b5e2c19aac5628d61b3f598aefe816d37e1b324df018c5dacf4cd4b" Oct 30 18:25:02.822650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.822623 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" event={"ID":"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e","Type":"ContainerDied","Data":"3a47235c7e61b1fe8c5b3590996f2ce2e23a04972f2c8cd2d9821ae7f94b92f5"} Oct 30 18:25:02.822650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.822652 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx" event={"ID":"ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e","Type":"ContainerDied","Data":"2a2b18286f1768f929a4e75002ae53aa83d0bb56061f0d30712fca9fcdc9aa72"} Oct 30 18:25:02.822823 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.822663 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" event={"ID":"f2ebd246-b899-4a1d-9067-3993742fb028","Type":"ContainerDied","Data":"aced874e598d3925ad113b36279d79937cb7ebbc84ed299c2493a0ff8f63e602"} Oct 30 18:25:02.822823 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.822672 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642" event={"ID":"f2ebd246-b899-4a1d-9067-3993742fb028","Type":"ContainerDied","Data":"deccf08beaa3e3870205db4707a6db43f33793f652559011c6a65b921ff9d320"} Oct 30 18:25:02.831013 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.830871 5232 scope.go:122] "RemoveContainer" containerID="197ba80f6b5e2c19aac5628d61b3f598aefe816d37e1b324df018c5dacf4cd4b" Oct 30 18:25:02.832081 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:25:02.832060 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"197ba80f6b5e2c19aac5628d61b3f598aefe816d37e1b324df018c5dacf4cd4b\": container with ID starting with 197ba80f6b5e2c19aac5628d61b3f598aefe816d37e1b324df018c5dacf4cd4b not found: ID does not exist" containerID="197ba80f6b5e2c19aac5628d61b3f598aefe816d37e1b324df018c5dacf4cd4b" Oct 30 18:25:02.832187 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.832091 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"197ba80f6b5e2c19aac5628d61b3f598aefe816d37e1b324df018c5dacf4cd4b"} err="failed to get container status \"197ba80f6b5e2c19aac5628d61b3f598aefe816d37e1b324df018c5dacf4cd4b\": rpc error: code = NotFound desc = could not find container \"197ba80f6b5e2c19aac5628d61b3f598aefe816d37e1b324df018c5dacf4cd4b\": container with ID starting with 197ba80f6b5e2c19aac5628d61b3f598aefe816d37e1b324df018c5dacf4cd4b not found: ID does not exist" Oct 30 18:25:02.832187 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.832110 5232 scope.go:122] "RemoveContainer" containerID="3a47235c7e61b1fe8c5b3590996f2ce2e23a04972f2c8cd2d9821ae7f94b92f5" Oct 30 18:25:02.842070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.842047 5232 scope.go:122] "RemoveContainer" containerID="18d030245395eb9dfb629d43f057cbbfd732280822d9d92f1debf184fe1e8296" Oct 30 18:25:02.853054 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.853040 5232 scope.go:122] "RemoveContainer" containerID="3a47235c7e61b1fe8c5b3590996f2ce2e23a04972f2c8cd2d9821ae7f94b92f5" Oct 30 18:25:02.853306 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:25:02.853286 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a47235c7e61b1fe8c5b3590996f2ce2e23a04972f2c8cd2d9821ae7f94b92f5\": container with ID starting with 3a47235c7e61b1fe8c5b3590996f2ce2e23a04972f2c8cd2d9821ae7f94b92f5 not found: ID does not exist" containerID="3a47235c7e61b1fe8c5b3590996f2ce2e23a04972f2c8cd2d9821ae7f94b92f5" Oct 30 18:25:02.853361 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.853312 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a47235c7e61b1fe8c5b3590996f2ce2e23a04972f2c8cd2d9821ae7f94b92f5"} err="failed to get container status \"3a47235c7e61b1fe8c5b3590996f2ce2e23a04972f2c8cd2d9821ae7f94b92f5\": rpc error: code = NotFound desc = could not find container \"3a47235c7e61b1fe8c5b3590996f2ce2e23a04972f2c8cd2d9821ae7f94b92f5\": container with ID starting with 3a47235c7e61b1fe8c5b3590996f2ce2e23a04972f2c8cd2d9821ae7f94b92f5 not found: ID does not exist" Oct 30 18:25:02.853361 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.853330 5232 scope.go:122] "RemoveContainer" containerID="18d030245395eb9dfb629d43f057cbbfd732280822d9d92f1debf184fe1e8296" Oct 30 18:25:02.853694 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:25:02.853671 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18d030245395eb9dfb629d43f057cbbfd732280822d9d92f1debf184fe1e8296\": container with ID starting with 18d030245395eb9dfb629d43f057cbbfd732280822d9d92f1debf184fe1e8296 not found: ID does not exist" containerID="18d030245395eb9dfb629d43f057cbbfd732280822d9d92f1debf184fe1e8296" Oct 30 18:25:02.853750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.853702 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18d030245395eb9dfb629d43f057cbbfd732280822d9d92f1debf184fe1e8296"} err="failed to get container status \"18d030245395eb9dfb629d43f057cbbfd732280822d9d92f1debf184fe1e8296\": rpc error: code = NotFound desc = could not find container \"18d030245395eb9dfb629d43f057cbbfd732280822d9d92f1debf184fe1e8296\": container with ID starting with 18d030245395eb9dfb629d43f057cbbfd732280822d9d92f1debf184fe1e8296 not found: ID does not exist" Oct 30 18:25:02.853750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.853716 5232 scope.go:122] "RemoveContainer" containerID="aced874e598d3925ad113b36279d79937cb7ebbc84ed299c2493a0ff8f63e602" Oct 30 18:25:02.884656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.884575 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt"] Oct 30 18:25:02.888479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.888444 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-9jqxt"] Oct 30 18:25:02.892590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.892573 5232 scope.go:122] "RemoveContainer" containerID="aced874e598d3925ad113b36279d79937cb7ebbc84ed299c2493a0ff8f63e602" Oct 30 18:25:02.892899 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:25:02.892882 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aced874e598d3925ad113b36279d79937cb7ebbc84ed299c2493a0ff8f63e602\": container with ID starting with aced874e598d3925ad113b36279d79937cb7ebbc84ed299c2493a0ff8f63e602 not found: ID does not exist" containerID="aced874e598d3925ad113b36279d79937cb7ebbc84ed299c2493a0ff8f63e602" Oct 30 18:25:02.892977 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.892905 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aced874e598d3925ad113b36279d79937cb7ebbc84ed299c2493a0ff8f63e602"} err="failed to get container status \"aced874e598d3925ad113b36279d79937cb7ebbc84ed299c2493a0ff8f63e602\": rpc error: code = NotFound desc = could not find container \"aced874e598d3925ad113b36279d79937cb7ebbc84ed299c2493a0ff8f63e602\": container with ID starting with aced874e598d3925ad113b36279d79937cb7ebbc84ed299c2493a0ff8f63e602 not found: ID does not exist" Oct 30 18:25:02.946280 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.946254 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx"] Oct 30 18:25:02.958448 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.958417 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-hrsmx"] Oct 30 18:25:02.994423 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.994394 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4"] Oct 30 18:25:02.998475 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:02.998440 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4"] Oct 30 18:25:03.033048 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:03.033021 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642"] Oct 30 18:25:03.036840 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:03.036814 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-q7642"] Oct 30 18:25:03.166991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:03.166908 5232 patch_prober.go:29] interesting pod/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:03.167131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:03.166986 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-ef13cc-openshift-default-5d997588b7-spsb4" podUID="8a806882-2d84-427c-8698-9ed54994b748" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:03.990140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:03.989863 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" podUID="158c3671-9513-4f20-bdca-2801ec139c14" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:25:03.990825 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:03.990796 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" podUID="964b7eb3-281f-4006-817e-6cb694f24dac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:25:04.817962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:04.817924 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7403bbb1-5c6e-431f-88a2-1005b4a22421" path="/var/lib/kubelet/pods/7403bbb1-5c6e-431f-88a2-1005b4a22421/volumes" Oct 30 18:25:04.818614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:04.818595 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a806882-2d84-427c-8698-9ed54994b748" path="/var/lib/kubelet/pods/8a806882-2d84-427c-8698-9ed54994b748/volumes" Oct 30 18:25:04.819214 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:04.819190 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f2ebd246-b899-4a1d-9067-3993742fb028" path="/var/lib/kubelet/pods/f2ebd246-b899-4a1d-9067-3993742fb028/volumes" Oct 30 18:25:04.820289 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:04.820267 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" path="/var/lib/kubelet/pods/ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e/volumes" Oct 30 18:25:06.536756 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.536723 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz"] Oct 30 18:25:06.537548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537530 5232 cpu_manager.go:414] "containerMap: removing container" podUID="546beae5-c98b-4b49-9bc8-46b77e64648c" containerName="revoke-keys" Oct 30 18:25:06.537548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537549 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="546beae5-c98b-4b49-9bc8-46b77e64648c" containerName="revoke-keys" Oct 30 18:25:06.537640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537568 5232 cpu_manager.go:414] "containerMap: removing container" podUID="8a806882-2d84-427c-8698-9ed54994b748" containerName="istio-proxy" Oct 30 18:25:06.537640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537572 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a806882-2d84-427c-8698-9ed54994b748" containerName="istio-proxy" Oct 30 18:25:06.537640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537587 5232 cpu_manager.go:414] "containerMap: removing container" podUID="ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" containerName="istio-proxy" Oct 30 18:25:06.537640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537592 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" containerName="istio-proxy" Oct 30 18:25:06.537640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537606 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7403bbb1-5c6e-431f-88a2-1005b4a22421" containerName="istio-proxy" Oct 30 18:25:06.537640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537611 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7403bbb1-5c6e-431f-88a2-1005b4a22421" containerName="istio-proxy" Oct 30 18:25:06.537640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537621 5232 cpu_manager.go:414] "containerMap: removing container" podUID="82285c8d-b803-4f2d-9ca7-ef302cf7b5fc" containerName="revoke-keys" Oct 30 18:25:06.537640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537626 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82285c8d-b803-4f2d-9ca7-ef302cf7b5fc" containerName="revoke-keys" Oct 30 18:25:06.537640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537632 5232 cpu_manager.go:414] "containerMap: removing container" podUID="f2ebd246-b899-4a1d-9067-3993742fb028" containerName="istio-proxy" Oct 30 18:25:06.537640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537636 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f2ebd246-b899-4a1d-9067-3993742fb028" containerName="istio-proxy" Oct 30 18:25:06.537923 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537654 5232 cpu_manager.go:414] "containerMap: removing container" podUID="ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" containerName="istio-proxy" Oct 30 18:25:06.537923 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537658 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" containerName="istio-proxy" Oct 30 18:25:06.537923 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537764 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" containerName="istio-proxy" Oct 30 18:25:06.537923 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537773 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7403bbb1-5c6e-431f-88a2-1005b4a22421" containerName="istio-proxy" Oct 30 18:25:06.537923 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537780 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="82285c8d-b803-4f2d-9ca7-ef302cf7b5fc" containerName="revoke-keys" Oct 30 18:25:06.537923 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537786 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="f2ebd246-b899-4a1d-9067-3993742fb028" containerName="istio-proxy" Oct 30 18:25:06.537923 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537793 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="ffe6b1f9-81fc-4d91-831e-7ddcb0668a1e" containerName="istio-proxy" Oct 30 18:25:06.537923 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537801 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="546beae5-c98b-4b49-9bc8-46b77e64648c" containerName="revoke-keys" Oct 30 18:25:06.537923 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.537808 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a806882-2d84-427c-8698-9ed54994b748" containerName="istio-proxy" Oct 30 18:25:06.543862 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.543841 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz" Oct 30 18:25:06.559731 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.559701 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz"] Oct 30 18:25:06.575893 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.575855 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:25:06.576063 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.575926 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:25:06.613326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.613297 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f8709f-wffcz\" (UID: \"86cc0758-fb7a-404f-9016-fd00ed2b6f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz" Oct 30 18:25:06.613636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.613618 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qzntg\" (UniqueName: \"kubernetes.io/projected/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-kube-api-access-qzntg\") pod \"maas-api-revoke-keys-e2e-api-a-f8709f-wffcz\" (UID: \"86cc0758-fb7a-404f-9016-fd00ed2b6f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz" Oct 30 18:25:06.613716 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.613705 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f8709f-wffcz\" (UID: \"86cc0758-fb7a-404f-9016-fd00ed2b6f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz" Oct 30 18:25:06.721540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.716413 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f8709f-wffcz\" (UID: \"86cc0758-fb7a-404f-9016-fd00ed2b6f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz" Oct 30 18:25:06.721540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.716739 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qzntg\" (UniqueName: \"kubernetes.io/projected/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-kube-api-access-qzntg\") pod \"maas-api-revoke-keys-e2e-api-a-f8709f-wffcz\" (UID: \"86cc0758-fb7a-404f-9016-fd00ed2b6f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz" Oct 30 18:25:06.721540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.716842 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f8709f-wffcz\" (UID: \"86cc0758-fb7a-404f-9016-fd00ed2b6f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz" Oct 30 18:25:06.721540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.717737 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f8709f-wffcz\" (UID: \"86cc0758-fb7a-404f-9016-fd00ed2b6f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz" Oct 30 18:25:06.743534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.728966 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f8709f-wffcz\" (UID: \"86cc0758-fb7a-404f-9016-fd00ed2b6f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz" Oct 30 18:25:06.743534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.742186 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qzntg\" (UniqueName: \"kubernetes.io/projected/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-kube-api-access-qzntg\") pod \"maas-api-revoke-keys-e2e-api-a-f8709f-wffcz\" (UID: \"86cc0758-fb7a-404f-9016-fd00ed2b6f4b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz" Oct 30 18:25:06.879593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:06.870153 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz" Oct 30 18:25:07.204651 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:25:07.204621 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod86cc0758_fb7a_404f_9016_fd00ed2b6f4b.slice/crio-c62354b9f51dfb5b305b9b5da2447f6d55317b354b9b790db09d9decfc3aaed6 WatchSource:0}: Error finding container c62354b9f51dfb5b305b9b5da2447f6d55317b354b9b790db09d9decfc3aaed6: Status 404 returned error can't find the container with id c62354b9f51dfb5b305b9b5da2447f6d55317b354b9b790db09d9decfc3aaed6 Oct 30 18:25:07.205453 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:07.205429 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz"] Oct 30 18:25:07.941804 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:07.941718 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz" event={"ID":"86cc0758-fb7a-404f-9016-fd00ed2b6f4b","Type":"ContainerStarted","Data":"67120584dae0790243e7e6a94063704bf18b6414b398e4c9a342526db1575c9b"} Oct 30 18:25:07.941804 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:07.941771 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz" event={"ID":"86cc0758-fb7a-404f-9016-fd00ed2b6f4b","Type":"ContainerStarted","Data":"c62354b9f51dfb5b305b9b5da2447f6d55317b354b9b790db09d9decfc3aaed6"} Oct 30 18:25:07.986857 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:07.986807 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz" podStartSLOduration=1.9867938600000001 podStartE2EDuration="1.98679386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:07.982761494 +0000 UTC m=+1407.299181817" watchObservedRunningTime="2026-10-30 18:25:07.98679386 +0000 UTC m=+1407.303214208" Oct 30 18:25:08.953125 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:08.953088 5232 generic.go:356] "Generic (PLEG): container finished" podID="86cc0758-fb7a-404f-9016-fd00ed2b6f4b" containerID="67120584dae0790243e7e6a94063704bf18b6414b398e4c9a342526db1575c9b" exitCode=0 Oct 30 18:25:08.953593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:08.953190 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz" event={"ID":"86cc0758-fb7a-404f-9016-fd00ed2b6f4b","Type":"ContainerDied","Data":"67120584dae0790243e7e6a94063704bf18b6414b398e4c9a342526db1575c9b"} Oct 30 18:25:10.136075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:10.136050 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz" Oct 30 18:25:10.216895 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:10.216809 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-maas-api-xks-ca\") pod \"86cc0758-fb7a-404f-9016-fd00ed2b6f4b\" (UID: \"86cc0758-fb7a-404f-9016-fd00ed2b6f4b\") " Oct 30 18:25:10.216895 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:10.216882 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-maas-api-openshift-ca\") pod \"86cc0758-fb7a-404f-9016-fd00ed2b6f4b\" (UID: \"86cc0758-fb7a-404f-9016-fd00ed2b6f4b\") " Oct 30 18:25:10.217290 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:10.217267 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-kube-api-access-qzntg\" (UniqueName: \"kubernetes.io/projected/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-kube-api-access-qzntg\") pod \"86cc0758-fb7a-404f-9016-fd00ed2b6f4b\" (UID: \"86cc0758-fb7a-404f-9016-fd00ed2b6f4b\") " Oct 30 18:25:10.217290 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:10.217269 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-maas-api-openshift-ca" pod "86cc0758-fb7a-404f-9016-fd00ed2b6f4b" (UID: "86cc0758-fb7a-404f-9016-fd00ed2b6f4b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:10.218219 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:10.218192 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:10.219273 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:10.219245 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-maas-api-xks-ca" pod "86cc0758-fb7a-404f-9016-fd00ed2b6f4b" (UID: "86cc0758-fb7a-404f-9016-fd00ed2b6f4b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:10.219628 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:10.219604 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-kube-api-access-qzntg" pod "86cc0758-fb7a-404f-9016-fd00ed2b6f4b" (UID: "86cc0758-fb7a-404f-9016-fd00ed2b6f4b"). InnerVolumeSpecName "kube-api-access-qzntg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:10.319232 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:10.319201 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qzntg\" (UniqueName: \"kubernetes.io/projected/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-kube-api-access-qzntg\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:10.319232 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:10.319230 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/86cc0758-fb7a-404f-9016-fd00ed2b6f4b-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:10.976254 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:10.976222 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz" Oct 30 18:25:10.976254 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:10.976228 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz" event={"ID":"86cc0758-fb7a-404f-9016-fd00ed2b6f4b","Type":"ContainerDied","Data":"c62354b9f51dfb5b305b9b5da2447f6d55317b354b9b790db09d9decfc3aaed6"} Oct 30 18:25:10.976501 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:10.976289 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c62354b9f51dfb5b305b9b5da2447f6d55317b354b9b790db09d9decfc3aaed6" Oct 30 18:25:13.958254 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:13.958213 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" podUID="964b7eb3-281f-4006-817e-6cb694f24dac" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:25:13.958695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:13.958222 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" podUID="158c3671-9513-4f20-bdca-2801ec139c14" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:25:17.179915 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:17.179887 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg"] Oct 30 18:25:17.210916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:17.210884 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq"] Oct 30 18:25:18.517537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:18.514581 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.688567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:18.681585 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.688567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:18.681742 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.749779 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:18.749730 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.771883 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:18.770605 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:18.884724 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:18.858635 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:20.404147 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.404116 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m"] Oct 30 18:25:20.406991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.405917 5232 cpu_manager.go:414] "containerMap: removing container" podUID="86cc0758-fb7a-404f-9016-fd00ed2b6f4b" containerName="revoke-keys" Oct 30 18:25:20.406991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.405937 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="86cc0758-fb7a-404f-9016-fd00ed2b6f4b" containerName="revoke-keys" Oct 30 18:25:20.406991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.406076 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="86cc0758-fb7a-404f-9016-fd00ed2b6f4b" containerName="revoke-keys" Oct 30 18:25:20.457536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.438848 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.457536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.441477 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m"] Oct 30 18:25:20.497897 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.489911 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-f0837a-openshift-default-dockercfg-5dz8z\"" Oct 30 18:25:20.620776 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.620726 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-workload-certs\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.620980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.620897 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-istio-data\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.620980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.620947 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-credential-socket\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.621099 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.621007 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-istio-envoy\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.621099 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.621087 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfpv8\" (UniqueName: \"kubernetes.io/projected/c5711415-04ea-4954-a672-2af1986aec4c-kube-api-access-zfpv8\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.621210 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.621133 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-workload-socket\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.621210 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.621175 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5711415-04ea-4954-a672-2af1986aec4c-istio-token\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.621314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.621241 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5711415-04ea-4954-a672-2af1986aec4c-istiod-ca-cert\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.621314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.621302 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5711415-04ea-4954-a672-2af1986aec4c-istio-podinfo\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.722752 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.722652 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfpv8\" (UniqueName: \"kubernetes.io/projected/c5711415-04ea-4954-a672-2af1986aec4c-kube-api-access-zfpv8\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.881631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.881598 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-workload-socket\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.882297 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.882273 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5711415-04ea-4954-a672-2af1986aec4c-istio-token\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.882856 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.882828 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5711415-04ea-4954-a672-2af1986aec4c-istiod-ca-cert\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.883087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.883065 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5711415-04ea-4954-a672-2af1986aec4c-istio-podinfo\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.891485 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.884438 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-workload-certs\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.892264 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.892242 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-istio-data\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.892750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.892731 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-credential-socket\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.893205 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.893188 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-istio-envoy\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.907394 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.889137 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-workload-certs\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.907665 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.882190 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-workload-socket\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.907906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.892650 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-istio-data\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.908085 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.815318 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfpv8\" (UniqueName: \"kubernetes.io/projected/c5711415-04ea-4954-a672-2af1986aec4c-kube-api-access-zfpv8\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.908244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.893094 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-credential-socket\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.908340 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.883867 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5711415-04ea-4954-a672-2af1986aec4c-istiod-ca-cert\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.950125 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.945738 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-istio-envoy\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.950963 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.950928 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5711415-04ea-4954-a672-2af1986aec4c-istio-token\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:20.951789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:20.951766 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5711415-04ea-4954-a672-2af1986aec4c-istio-podinfo\") pod \"e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:21.193552 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:21.192788 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:21.700795 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:21.700749 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:22.213542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:22.209755 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg" podUID="60ace843-6055-4728-a15f-6c6f65c449c3" containerName="payload-processing" containerID="cri-o://18aee18810fc882e5ca7e16d2acc6fb558eded02d68e1ab8f4289b943583aca0" gracePeriod=25 Oct 30 18:25:22.223026 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:22.222290 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq" podUID="4cf43950-0cdf-489c-9fd6-95ef1bb93cea" containerName="payload-pre-processing" containerID="cri-o://413eaa14c676bef4fb87b40157921525c83f853188c490f07bf5c382dbb1f3d9" gracePeriod=25 Oct 30 18:25:22.301388 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:22.301354 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m"] Oct 30 18:25:22.407332 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:22.407071 5232 generic.go:356] "Generic (PLEG): container finished" podID="964b7eb3-281f-4006-817e-6cb694f24dac" containerID="6f6ac0aa1ce4aed817bc78764f91776383b136572ea5987028f524f6f324c2d4" exitCode=0 Oct 30 18:25:22.407332 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:22.407250 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" event={"ID":"964b7eb3-281f-4006-817e-6cb694f24dac","Type":"ContainerDied","Data":"6f6ac0aa1ce4aed817bc78764f91776383b136572ea5987028f524f6f324c2d4"} Oct 30 18:25:22.430927 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:22.430790 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:25:22.431270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:22.431230 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:25:22.431471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:22.431436 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:25:22.504900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:22.504865 5232 generic.go:356] "Generic (PLEG): container finished" podID="158c3671-9513-4f20-bdca-2801ec139c14" containerID="1bb119c347160d9a63c813acb5d09ec9ff19ac59fb9624e8779d42973c717d44" exitCode=0 Oct 30 18:25:22.505042 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:22.504903 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" event={"ID":"158c3671-9513-4f20-bdca-2801ec139c14","Type":"ContainerDied","Data":"1bb119c347160d9a63c813acb5d09ec9ff19ac59fb9624e8779d42973c717d44"} Oct 30 18:25:23.151891 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:23.120736 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" Oct 30 18:25:23.204070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:23.193081 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" Oct 30 18:25:23.204070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:23.202002 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/964b7eb3-281f-4006-817e-6cb694f24dac-kube-api-access-h4hjz\" (UniqueName: \"kubernetes.io/projected/964b7eb3-281f-4006-817e-6cb694f24dac-kube-api-access-h4hjz\") pod \"964b7eb3-281f-4006-817e-6cb694f24dac\" (UID: \"964b7eb3-281f-4006-817e-6cb694f24dac\") " Oct 30 18:25:23.204070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:23.202160 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/964b7eb3-281f-4006-817e-6cb694f24dac-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964b7eb3-281f-4006-817e-6cb694f24dac-plugins-config-volume\") pod \"964b7eb3-281f-4006-817e-6cb694f24dac\" (UID: \"964b7eb3-281f-4006-817e-6cb694f24dac\") " Oct 30 18:25:23.252148 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:23.234335 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/964b7eb3-281f-4006-817e-6cb694f24dac-kube-api-access-h4hjz" pod "964b7eb3-281f-4006-817e-6cb694f24dac" (UID: "964b7eb3-281f-4006-817e-6cb694f24dac"). InnerVolumeSpecName "kube-api-access-h4hjz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:23.358590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:23.309683 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h4hjz\" (UniqueName: \"kubernetes.io/projected/964b7eb3-281f-4006-817e-6cb694f24dac-kube-api-access-h4hjz\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:23.510622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:23.506108 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/964b7eb3-281f-4006-817e-6cb694f24dac-plugins-config-volume" pod "964b7eb3-281f-4006-817e-6cb694f24dac" (UID: "964b7eb3-281f-4006-817e-6cb694f24dac"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:23.690722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:23.690487 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/158c3671-9513-4f20-bdca-2801ec139c14-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158c3671-9513-4f20-bdca-2801ec139c14-plugins-config-volume\") pod \"158c3671-9513-4f20-bdca-2801ec139c14\" (UID: \"158c3671-9513-4f20-bdca-2801ec139c14\") " Oct 30 18:25:23.690722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:23.690570 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/158c3671-9513-4f20-bdca-2801ec139c14-kube-api-access-86tqt\" (UniqueName: \"kubernetes.io/projected/158c3671-9513-4f20-bdca-2801ec139c14-kube-api-access-86tqt\") pod \"158c3671-9513-4f20-bdca-2801ec139c14\" (UID: \"158c3671-9513-4f20-bdca-2801ec139c14\") " Oct 30 18:25:23.905250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:23.905170 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/158c3671-9513-4f20-bdca-2801ec139c14-plugins-config-volume" pod "158c3671-9513-4f20-bdca-2801ec139c14" (UID: "158c3671-9513-4f20-bdca-2801ec139c14"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:23.906608 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:23.906286 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/158c3671-9513-4f20-bdca-2801ec139c14-kube-api-access-86tqt" pod "158c3671-9513-4f20-bdca-2801ec139c14" (UID: "158c3671-9513-4f20-bdca-2801ec139c14"). InnerVolumeSpecName "kube-api-access-86tqt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:23.960624 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:23.955131 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/964b7eb3-281f-4006-817e-6cb694f24dac-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:24.003801 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:23.961962 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" event={"ID":"964b7eb3-281f-4006-817e-6cb694f24dac","Type":"ContainerDied","Data":"9739268a2b1709c76931a8314191315e3369463d981b51b364784f343d99f9ee"} Oct 30 18:25:24.003801 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:23.962032 5232 scope.go:122] "RemoveContainer" containerID="6f6ac0aa1ce4aed817bc78764f91776383b136572ea5987028f524f6f324c2d4" Oct 30 18:25:24.003801 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:23.962217 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9" Oct 30 18:25:24.175188 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.146989 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/158c3671-9513-4f20-bdca-2801ec139c14-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:24.175188 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.147025 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86tqt\" (UniqueName: \"kubernetes.io/projected/158c3671-9513-4f20-bdca-2801ec139c14-kube-api-access-86tqt\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:24.282431 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.273689 5232 generic.go:356] "Generic (PLEG): container finished" podID="60ace843-6055-4728-a15f-6c6f65c449c3" containerID="18aee18810fc882e5ca7e16d2acc6fb558eded02d68e1ab8f4289b943583aca0" exitCode=0 Oct 30 18:25:24.282431 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.273905 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg" event={"ID":"60ace843-6055-4728-a15f-6c6f65c449c3","Type":"ContainerDied","Data":"18aee18810fc882e5ca7e16d2acc6fb558eded02d68e1ab8f4289b943583aca0"} Oct 30 18:25:24.408412 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.408376 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" event={"ID":"c5711415-04ea-4954-a672-2af1986aec4c","Type":"ContainerStarted","Data":"e274ceddbbde51399c65f2b46e78a8132cec3ee8400152bd462e8728ea7cc3fe"} Oct 30 18:25:24.409355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.409291 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" event={"ID":"c5711415-04ea-4954-a672-2af1986aec4c","Type":"ContainerStarted","Data":"b7405fc941451fb3dedb595667c8a5935ddbedc6ea1be3858f9ac30a60ed21ac"} Oct 30 18:25:24.429617 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.413673 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq" Oct 30 18:25:24.453545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.443716 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg" Oct 30 18:25:24.453725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.453644 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:24.480082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.480039 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60ace843-6055-4728-a15f-6c6f65c449c3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ace843-6055-4728-a15f-6c6f65c449c3-plugins-config-volume\") pod \"60ace843-6055-4728-a15f-6c6f65c449c3\" (UID: \"60ace843-6055-4728-a15f-6c6f65c449c3\") " Oct 30 18:25:24.487034 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.487006 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4cf43950-0cdf-489c-9fd6-95ef1bb93cea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf43950-0cdf-489c-9fd6-95ef1bb93cea-plugins-config-volume\") pod \"4cf43950-0cdf-489c-9fd6-95ef1bb93cea\" (UID: \"4cf43950-0cdf-489c-9fd6-95ef1bb93cea\") " Oct 30 18:25:24.488709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.488679 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4cf43950-0cdf-489c-9fd6-95ef1bb93cea-kube-api-access-smhp5\" (UniqueName: \"kubernetes.io/projected/4cf43950-0cdf-489c-9fd6-95ef1bb93cea-kube-api-access-smhp5\") pod \"4cf43950-0cdf-489c-9fd6-95ef1bb93cea\" (UID: \"4cf43950-0cdf-489c-9fd6-95ef1bb93cea\") " Oct 30 18:25:24.489089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.489054 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60ace843-6055-4728-a15f-6c6f65c449c3-kube-api-access-9cncf\" (UniqueName: \"kubernetes.io/projected/60ace843-6055-4728-a15f-6c6f65c449c3-kube-api-access-9cncf\") pod \"60ace843-6055-4728-a15f-6c6f65c449c3\" (UID: \"60ace843-6055-4728-a15f-6c6f65c449c3\") " Oct 30 18:25:24.490939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.480751 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60ace843-6055-4728-a15f-6c6f65c449c3-plugins-config-volume" pod "60ace843-6055-4728-a15f-6c6f65c449c3" (UID: "60ace843-6055-4728-a15f-6c6f65c449c3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:24.493252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.488285 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4cf43950-0cdf-489c-9fd6-95ef1bb93cea-plugins-config-volume" pod "4cf43950-0cdf-489c-9fd6-95ef1bb93cea" (UID: "4cf43950-0cdf-489c-9fd6-95ef1bb93cea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:24.505955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.505923 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" event={"ID":"158c3671-9513-4f20-bdca-2801ec139c14","Type":"ContainerDied","Data":"845d27504d5035330d4bb169942f111dac30dccb22f5f3dcf8f1dc7385a53f11"} Oct 30 18:25:24.506093 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.506047 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b" Oct 30 18:25:24.508634 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.508596 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4cf43950-0cdf-489c-9fd6-95ef1bb93cea-kube-api-access-smhp5" pod "4cf43950-0cdf-489c-9fd6-95ef1bb93cea" (UID: "4cf43950-0cdf-489c-9fd6-95ef1bb93cea"). InnerVolumeSpecName "kube-api-access-smhp5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:24.587670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.573566 5232 generic.go:356] "Generic (PLEG): container finished" podID="4cf43950-0cdf-489c-9fd6-95ef1bb93cea" containerID="413eaa14c676bef4fb87b40157921525c83f853188c490f07bf5c382dbb1f3d9" exitCode=0 Oct 30 18:25:24.587670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.573729 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq" event={"ID":"4cf43950-0cdf-489c-9fd6-95ef1bb93cea","Type":"ContainerDied","Data":"413eaa14c676bef4fb87b40157921525c83f853188c490f07bf5c382dbb1f3d9"} Oct 30 18:25:24.587670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.573838 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq" Oct 30 18:25:24.631687 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.610472 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60ace843-6055-4728-a15f-6c6f65c449c3-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:24.631687 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.610526 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4cf43950-0cdf-489c-9fd6-95ef1bb93cea-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:24.631687 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.610544 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-smhp5\" (UniqueName: \"kubernetes.io/projected/4cf43950-0cdf-489c-9fd6-95ef1bb93cea-kube-api-access-smhp5\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:24.631687 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.610635 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9"] Oct 30 18:25:24.636452 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.636420 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-fdf4da-69ff95b66c-jrhh9"] Oct 30 18:25:24.671609 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.665263 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" podStartSLOduration=4.6652392670000005 podStartE2EDuration="4.665239267s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:24.570197633 +0000 UTC m=+1423.886617955" watchObservedRunningTime="2026-10-30 18:25:24.665239267 +0000 UTC m=+1423.981659588" Oct 30 18:25:24.718849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.718805 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60ace843-6055-4728-a15f-6c6f65c449c3-kube-api-access-9cncf" pod "60ace843-6055-4728-a15f-6c6f65c449c3" (UID: "60ace843-6055-4728-a15f-6c6f65c449c3"). InnerVolumeSpecName "kube-api-access-9cncf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:24.860670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:24.820409 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9cncf\" (UniqueName: \"kubernetes.io/projected/60ace843-6055-4728-a15f-6c6f65c449c3-kube-api-access-9cncf\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:25.022537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:25.006926 5232 scope.go:122] "RemoveContainer" containerID="1bb119c347160d9a63c813acb5d09ec9ff19ac59fb9624e8779d42973c717d44" Oct 30 18:25:25.106787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:25.106143 5232 scope.go:122] "RemoveContainer" containerID="413eaa14c676bef4fb87b40157921525c83f853188c490f07bf5c382dbb1f3d9" Oct 30 18:25:25.164539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:25.109917 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="964b7eb3-281f-4006-817e-6cb694f24dac" path="/var/lib/kubelet/pods/964b7eb3-281f-4006-817e-6cb694f24dac/volumes" Oct 30 18:25:25.166218 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:25.165755 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq"] Oct 30 18:25:25.166218 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:25.165798 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f8709f-8676d4ff-hv4wq"] Oct 30 18:25:25.166218 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:25.165820 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b"] Oct 30 18:25:25.172431 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:25.172402 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-fdf4da-55966b9c98-knj7b"] Oct 30 18:25:25.194056 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:25.194030 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:25.199662 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:25.199553 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:25.587655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:25.587621 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg" Oct 30 18:25:25.588009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:25.587617 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg" event={"ID":"60ace843-6055-4728-a15f-6c6f65c449c3","Type":"ContainerDied","Data":"76a2c74be401b703a3f465d5ab43f9b728001c0fd587a7dab74bf52342ab07f8"} Oct 30 18:25:25.588128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:25.588081 5232 scope.go:122] "RemoveContainer" containerID="18aee18810fc882e5ca7e16d2acc6fb558eded02d68e1ab8f4289b943583aca0" Oct 30 18:25:25.592398 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:25.592371 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:25.593666 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:25.593634 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:25:25.694579 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:25.694438 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg"] Oct 30 18:25:25.701936 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:25.701895 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f8709f-b4dfdd89d-kgrxg"] Oct 30 18:25:26.815364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:26.814751 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="158c3671-9513-4f20-bdca-2801ec139c14" path="/var/lib/kubelet/pods/158c3671-9513-4f20-bdca-2801ec139c14/volumes" Oct 30 18:25:26.815894 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:26.815477 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4cf43950-0cdf-489c-9fd6-95ef1bb93cea" path="/var/lib/kubelet/pods/4cf43950-0cdf-489c-9fd6-95ef1bb93cea/volumes" Oct 30 18:25:26.818530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:26.816288 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60ace843-6055-4728-a15f-6c6f65c449c3" path="/var/lib/kubelet/pods/60ace843-6055-4728-a15f-6c6f65c449c3/volumes" Oct 30 18:25:27.155899 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:25:27.155809 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:25:27.768822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:27.768784 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh"] Oct 30 18:25:27.770036 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:27.770005 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" podUID="a767e3a9-c897-44b6-87c8-b1be86f89ca7" containerName="istio-proxy" containerID="cri-o://baad1f2a1aa15aa2aaa41c9ea568d972515fb1f520382e3ef537e3a59c7f81f2" gracePeriod=30 Oct 30 18:25:28.093785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:28.093688 5232 patch_prober.go:29] interesting pod/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:25:28.093785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:28.093758 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" podUID="7dbbd730-9357-4f2f-b2d3-f41c877205e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:29.457723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.457641 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh"] Oct 30 18:25:29.458362 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.458344 5232 cpu_manager.go:414] "containerMap: removing container" podUID="158c3671-9513-4f20-bdca-2801ec139c14" containerName="payload-processing" Oct 30 18:25:29.458362 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.458361 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="158c3671-9513-4f20-bdca-2801ec139c14" containerName="payload-processing" Oct 30 18:25:29.458565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.458385 5232 cpu_manager.go:414] "containerMap: removing container" podUID="4cf43950-0cdf-489c-9fd6-95ef1bb93cea" containerName="payload-pre-processing" Oct 30 18:25:29.458565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.458391 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4cf43950-0cdf-489c-9fd6-95ef1bb93cea" containerName="payload-pre-processing" Oct 30 18:25:29.458565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.458415 5232 cpu_manager.go:414] "containerMap: removing container" podUID="964b7eb3-281f-4006-817e-6cb694f24dac" containerName="payload-pre-processing" Oct 30 18:25:29.458565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.458420 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="964b7eb3-281f-4006-817e-6cb694f24dac" containerName="payload-pre-processing" Oct 30 18:25:29.458565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.458435 5232 cpu_manager.go:414] "containerMap: removing container" podUID="60ace843-6055-4728-a15f-6c6f65c449c3" containerName="payload-processing" Oct 30 18:25:29.458565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.458440 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60ace843-6055-4728-a15f-6c6f65c449c3" containerName="payload-processing" Oct 30 18:25:29.458895 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.458602 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="964b7eb3-281f-4006-817e-6cb694f24dac" containerName="payload-pre-processing" Oct 30 18:25:29.458895 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.458617 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="60ace843-6055-4728-a15f-6c6f65c449c3" containerName="payload-processing" Oct 30 18:25:29.458895 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.458633 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="158c3671-9513-4f20-bdca-2801ec139c14" containerName="payload-processing" Oct 30 18:25:29.458895 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.458642 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="4cf43950-0cdf-489c-9fd6-95ef1bb93cea" containerName="payload-pre-processing" Oct 30 18:25:29.477862 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.477820 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh" Oct 30 18:25:29.482452 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.482425 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh"] Oct 30 18:25:29.493496 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.493374 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w4-f0837a-dockercfg-r7d9n\"" Oct 30 18:25:29.505848 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.505817 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-f0837a\"" Oct 30 18:25:29.717581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.684868 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca962a0-f12b-43a2-994e-43a21e407dd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh\" (UID: \"2ca962a0-f12b-43a2-994e-43a21e407dd2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh" Oct 30 18:25:29.717581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.711488 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cnzkh\" (UniqueName: \"kubernetes.io/projected/2ca962a0-f12b-43a2-994e-43a21e407dd2-kube-api-access-cnzkh\") pod \"payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh\" (UID: \"2ca962a0-f12b-43a2-994e-43a21e407dd2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh" Oct 30 18:25:29.717581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.712087 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb"] Oct 30 18:25:29.945110 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.822255 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca962a0-f12b-43a2-994e-43a21e407dd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh\" (UID: \"2ca962a0-f12b-43a2-994e-43a21e407dd2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh" Oct 30 18:25:29.945110 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.822306 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cnzkh\" (UniqueName: \"kubernetes.io/projected/2ca962a0-f12b-43a2-994e-43a21e407dd2-kube-api-access-cnzkh\") pod \"payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh\" (UID: \"2ca962a0-f12b-43a2-994e-43a21e407dd2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh" Oct 30 18:25:29.945110 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.918797 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb"] Oct 30 18:25:29.945110 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.921688 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb" Oct 30 18:25:29.948605 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.948533 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca962a0-f12b-43a2-994e-43a21e407dd2-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh\" (UID: \"2ca962a0-f12b-43a2-994e-43a21e407dd2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh" Oct 30 18:25:29.985093 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:29.984096 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cnzkh\" (UniqueName: \"kubernetes.io/projected/2ca962a0-f12b-43a2-994e-43a21e407dd2-kube-api-access-cnzkh\") pod \"payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh\" (UID: \"2ca962a0-f12b-43a2-994e-43a21e407dd2\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh" Oct 30 18:25:30.038955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:30.038920 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txwhl\" (UniqueName: \"kubernetes.io/projected/ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b-kube-api-access-txwhl\") pod \"payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb\" (UID: \"ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb" Oct 30 18:25:30.039138 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:30.038972 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb\" (UID: \"ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb" Oct 30 18:25:30.111585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:30.111545 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh" Oct 30 18:25:30.140914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:30.140881 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txwhl\" (UniqueName: \"kubernetes.io/projected/ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b-kube-api-access-txwhl\") pod \"payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb\" (UID: \"ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb" Oct 30 18:25:30.141083 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:30.140943 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb\" (UID: \"ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb" Oct 30 18:25:30.142245 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:30.142216 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb\" (UID: \"ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb" Oct 30 18:25:30.163220 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:30.163142 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txwhl\" (UniqueName: \"kubernetes.io/projected/ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b-kube-api-access-txwhl\") pod \"payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb\" (UID: \"ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb" Oct 30 18:25:30.355352 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:30.355321 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb" Oct 30 18:25:30.465334 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:25:30.465298 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ca962a0_f12b_43a2_994e_43a21e407dd2.slice/crio-2200810e1447b954bfbfd4fafe4ed345c57efdc6a24cf818c3dd16226ed2967c WatchSource:0}: Error finding container 2200810e1447b954bfbfd4fafe4ed345c57efdc6a24cf818c3dd16226ed2967c: Status 404 returned error can't find the container with id 2200810e1447b954bfbfd4fafe4ed345c57efdc6a24cf818c3dd16226ed2967c Oct 30 18:25:30.468093 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:30.468067 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh"] Oct 30 18:25:30.732741 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:30.732711 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb"] Oct 30 18:25:30.733721 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:25:30.733686 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec1e9fc9_981d_48fc_8f19_6e8dcdc06d3b.slice/crio-9f47c464d5b026d969405117ba6bedec2c19a70a16f6fade4cd9bb688e8a8a19 WatchSource:0}: Error finding container 9f47c464d5b026d969405117ba6bedec2c19a70a16f6fade4cd9bb688e8a8a19: Status 404 returned error can't find the container with id 9f47c464d5b026d969405117ba6bedec2c19a70a16f6fade4cd9bb688e8a8a19 Oct 30 18:25:30.916829 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:30.916742 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh" event={"ID":"2ca962a0-f12b-43a2-994e-43a21e407dd2","Type":"ContainerStarted","Data":"8a4220c23b4a9f9ecf4279c14f1d1c2e8a87a47d8de57ade9e384829c43ce22d"} Oct 30 18:25:30.916829 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:30.916804 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh" event={"ID":"2ca962a0-f12b-43a2-994e-43a21e407dd2","Type":"ContainerStarted","Data":"2200810e1447b954bfbfd4fafe4ed345c57efdc6a24cf818c3dd16226ed2967c"} Oct 30 18:25:30.916829 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:30.916829 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh" Oct 30 18:25:31.524065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:30.918869 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb" event={"ID":"ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b","Type":"ContainerStarted","Data":"bcf0d99a0e043328883f081b2c8e83836a1746e488e8a5549f479011168aa723"} Oct 30 18:25:31.524065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:30.918909 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb" event={"ID":"ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b","Type":"ContainerStarted","Data":"9f47c464d5b026d969405117ba6bedec2c19a70a16f6fade4cd9bb688e8a8a19"} Oct 30 18:25:31.524065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:30.919059 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb" Oct 30 18:25:31.524065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:30.960186 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh" podStartSLOduration=1.9601669579999998 podStartE2EDuration="1.960166958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:30.959726826 +0000 UTC m=+1430.276147167" watchObservedRunningTime="2026-10-30 18:25:30.960166958 +0000 UTC m=+1430.276587280" Oct 30 18:25:31.524065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:31.003362 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb" podStartSLOduration=2.003348482 podStartE2EDuration="2.003348482s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:31.000176507 +0000 UTC m=+1430.316596867" watchObservedRunningTime="2026-10-30 18:25:31.003348482 +0000 UTC m=+1430.319768802" Oct 30 18:25:31.929294 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:31.929266 5232 generic.go:356] "Generic (PLEG): container finished" podID="7dbbd730-9357-4f2f-b2d3-f41c877205e3" containerID="ef915c07cd6bb50f797e5174e691fd7981623d37390a50fc7a2934a335142f26" exitCode=0 Oct 30 18:25:31.929483 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:31.929336 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" event={"ID":"7dbbd730-9357-4f2f-b2d3-f41c877205e3","Type":"ContainerDied","Data":"ef915c07cd6bb50f797e5174e691fd7981623d37390a50fc7a2934a335142f26"} Oct 30 18:25:31.930083 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:31.930064 5232 scope.go:122] "RemoveContainer" containerID="ef915c07cd6bb50f797e5174e691fd7981623d37390a50fc7a2934a335142f26" Oct 30 18:25:31.931684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:31.931655 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:25:31.931766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:31.931721 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:25:31.931766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:31.931756 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:25:32.991285 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:32.991231 5232 generic.go:356] "Generic (PLEG): container finished" podID="961c8dee-4505-4b84-b984-3006b6a95dd8" containerID="615e3e5a46e56b54b6f6be9d6d6d7cccaa5dc27727a25838ad2b231bf448ab3e" exitCode=0 Oct 30 18:25:32.991755 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:32.991362 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" event={"ID":"961c8dee-4505-4b84-b984-3006b6a95dd8","Type":"ContainerDied","Data":"615e3e5a46e56b54b6f6be9d6d6d7cccaa5dc27727a25838ad2b231bf448ab3e"} Oct 30 18:25:32.991956 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:32.991936 5232 scope.go:122] "RemoveContainer" containerID="615e3e5a46e56b54b6f6be9d6d6d7cccaa5dc27727a25838ad2b231bf448ab3e" Oct 30 18:25:32.995378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:32.995342 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:25:33.003988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:32.997696 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:25:33.003988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:32.997761 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:25:33.003988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:32.995385 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" event={"ID":"7dbbd730-9357-4f2f-b2d3-f41c877205e3","Type":"ContainerStarted","Data":"8875f64cfc317a51af45be8c0441d7bef0a6ad9e90277f9a219aeec6b1167ade"} Oct 30 18:25:33.003988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.001170 5232 generic.go:356] "Generic (PLEG): container finished" podID="7b16e19a-e62c-47b4-b695-ccc7164524b5" containerID="dc88d7c05e156b2e31e11cdf0e8850164c98e39a3b42d279cb811095a855ec7d" exitCode=0 Oct 30 18:25:33.003988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.001281 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" event={"ID":"7b16e19a-e62c-47b4-b695-ccc7164524b5","Type":"ContainerDied","Data":"dc88d7c05e156b2e31e11cdf0e8850164c98e39a3b42d279cb811095a855ec7d"} Oct 30 18:25:33.003988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.001676 5232 scope.go:122] "RemoveContainer" containerID="dc88d7c05e156b2e31e11cdf0e8850164c98e39a3b42d279cb811095a855ec7d" Oct 30 18:25:33.006364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.005977 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:25:33.006364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.006043 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:25:33.006364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.006077 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:25:33.398794 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.398771 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:25:33.571538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.568474 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-podinfo\") pod \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " Oct 30 18:25:33.571538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.568587 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-data\") pod \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " Oct 30 18:25:33.571538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.568663 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-workload-certs\") pod \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " Oct 30 18:25:33.571538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.568752 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-token\" (UniqueName: \"kubernetes.io/projected/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-token\") pod \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " Oct 30 18:25:33.571538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.568803 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-workload-socket\") pod \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " Oct 30 18:25:33.571538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.568886 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-credential-socket\") pod \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " Oct 30 18:25:33.571538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.568935 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istiod-ca-cert\") pod \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " Oct 30 18:25:33.571538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.569139 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a767e3a9-c897-44b6-87c8-b1be86f89ca7-kube-api-access-fx7gl\" (UniqueName: \"kubernetes.io/projected/a767e3a9-c897-44b6-87c8-b1be86f89ca7-kube-api-access-fx7gl\") pod \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " Oct 30 18:25:33.571538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.569227 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-envoy\") pod \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\" (UID: \"a767e3a9-c897-44b6-87c8-b1be86f89ca7\") " Oct 30 18:25:33.583034 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.581335 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-envoy" pod "a767e3a9-c897-44b6-87c8-b1be86f89ca7" (UID: "a767e3a9-c897-44b6-87c8-b1be86f89ca7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:33.583034 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.582746 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istiod-ca-cert" pod "a767e3a9-c897-44b6-87c8-b1be86f89ca7" (UID: "a767e3a9-c897-44b6-87c8-b1be86f89ca7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:33.585951 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.585759 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-workload-socket" pod "a767e3a9-c897-44b6-87c8-b1be86f89ca7" (UID: "a767e3a9-c897-44b6-87c8-b1be86f89ca7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:33.586096 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.585997 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-workload-certs" pod "a767e3a9-c897-44b6-87c8-b1be86f89ca7" (UID: "a767e3a9-c897-44b6-87c8-b1be86f89ca7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:33.586732 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.586224 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-credential-socket" pod "a767e3a9-c897-44b6-87c8-b1be86f89ca7" (UID: "a767e3a9-c897-44b6-87c8-b1be86f89ca7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:33.586732 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.586681 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a767e3a9-c897-44b6-87c8-b1be86f89ca7-kube-api-access-fx7gl" pod "a767e3a9-c897-44b6-87c8-b1be86f89ca7" (UID: "a767e3a9-c897-44b6-87c8-b1be86f89ca7"). InnerVolumeSpecName "kube-api-access-fx7gl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:33.586984 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.586951 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-data" pod "a767e3a9-c897-44b6-87c8-b1be86f89ca7" (UID: "a767e3a9-c897-44b6-87c8-b1be86f89ca7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:33.587909 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.587874 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-token" pod "a767e3a9-c897-44b6-87c8-b1be86f89ca7" (UID: "a767e3a9-c897-44b6-87c8-b1be86f89ca7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:33.587909 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.587889 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-podinfo" pod "a767e3a9-c897-44b6-87c8-b1be86f89ca7" (UID: "a767e3a9-c897-44b6-87c8-b1be86f89ca7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:25:33.672345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.672255 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fx7gl\" (UniqueName: \"kubernetes.io/projected/a767e3a9-c897-44b6-87c8-b1be86f89ca7-kube-api-access-fx7gl\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.672345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.672283 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.672345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.672296 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.672345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.672304 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.672345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.672315 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.672345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.672323 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.672345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.672332 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.672345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.672340 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a767e3a9-c897-44b6-87c8-b1be86f89ca7-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.672345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.672350 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a767e3a9-c897-44b6-87c8-b1be86f89ca7-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:33.968322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.966852 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:25:33.968322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.967533 5232 patch_prober.go:29] interesting pod/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 18:25:33.968322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:33.967610 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" podUID="7dbbd730-9357-4f2f-b2d3-f41c877205e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 18:25:34.076534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.069993 5232 generic.go:356] "Generic (PLEG): container finished" podID="a767e3a9-c897-44b6-87c8-b1be86f89ca7" containerID="baad1f2a1aa15aa2aaa41c9ea568d972515fb1f520382e3ef537e3a59c7f81f2" exitCode=0 Oct 30 18:25:34.076534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.070151 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" event={"ID":"a767e3a9-c897-44b6-87c8-b1be86f89ca7","Type":"ContainerDied","Data":"baad1f2a1aa15aa2aaa41c9ea568d972515fb1f520382e3ef537e3a59c7f81f2"} Oct 30 18:25:34.076534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.070208 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" event={"ID":"a767e3a9-c897-44b6-87c8-b1be86f89ca7","Type":"ContainerDied","Data":"5295c49e30aec65c2ada0cd00f2a947c76d789ee2dbafee417a0f3d9fb30b401"} Oct 30 18:25:34.076534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.070242 5232 scope.go:122] "RemoveContainer" containerID="baad1f2a1aa15aa2aaa41c9ea568d972515fb1f520382e3ef537e3a59c7f81f2" Oct 30 18:25:34.076534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.070432 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh" Oct 30 18:25:34.100245 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.097007 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" event={"ID":"961c8dee-4505-4b84-b984-3006b6a95dd8","Type":"ContainerStarted","Data":"53ff8a18a2e8a3e875aa6adcd6d5c9c11df3a5ea338f3151839b5b7e1ba28ebd"} Oct 30 18:25:34.102679 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.102651 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" event={"ID":"7b16e19a-e62c-47b4-b695-ccc7164524b5","Type":"ContainerStarted","Data":"887adb3ce5d62083c4399327087dabba8ad5131dfda0bd3082011f56093cc782"} Oct 30 18:25:34.165734 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.165699 5232 scope.go:122] "RemoveContainer" containerID="baad1f2a1aa15aa2aaa41c9ea568d972515fb1f520382e3ef537e3a59c7f81f2" Oct 30 18:25:34.166558 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:25:34.166531 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"baad1f2a1aa15aa2aaa41c9ea568d972515fb1f520382e3ef537e3a59c7f81f2\": container with ID starting with baad1f2a1aa15aa2aaa41c9ea568d972515fb1f520382e3ef537e3a59c7f81f2 not found: ID does not exist" containerID="baad1f2a1aa15aa2aaa41c9ea568d972515fb1f520382e3ef537e3a59c7f81f2" Oct 30 18:25:34.166740 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.166714 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"baad1f2a1aa15aa2aaa41c9ea568d972515fb1f520382e3ef537e3a59c7f81f2"} err="failed to get container status \"baad1f2a1aa15aa2aaa41c9ea568d972515fb1f520382e3ef537e3a59c7f81f2\": rpc error: code = NotFound desc = could not find container \"baad1f2a1aa15aa2aaa41c9ea568d972515fb1f520382e3ef537e3a59c7f81f2\": container with ID starting with baad1f2a1aa15aa2aaa41c9ea568d972515fb1f520382e3ef537e3a59c7f81f2 not found: ID does not exist" Oct 30 18:25:34.291611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.283355 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:25:34.291611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.283577 5232 patch_prober.go:29] interesting pod/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 18:25:34.291611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.283632 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" podUID="961c8dee-4505-4b84-b984-3006b6a95dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 18:25:34.297132 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.297093 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh"] Oct 30 18:25:34.301337 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.301310 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-vvsqh"] Oct 30 18:25:34.706638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.706558 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc"] Oct 30 18:25:34.707691 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.707668 5232 cpu_manager.go:414] "containerMap: removing container" podUID="a767e3a9-c897-44b6-87c8-b1be86f89ca7" containerName="istio-proxy" Oct 30 18:25:34.707756 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.707691 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a767e3a9-c897-44b6-87c8-b1be86f89ca7" containerName="istio-proxy" Oct 30 18:25:34.707877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.707863 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="a767e3a9-c897-44b6-87c8-b1be86f89ca7" containerName="istio-proxy" Oct 30 18:25:34.751794 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.751741 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc"] Oct 30 18:25:34.751972 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.751888 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc" Oct 30 18:25:34.841550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.820120 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjqr2\" (UniqueName: \"kubernetes.io/projected/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-kube-api-access-kjqr2\") pod \"maas-api-revoke-keys-e2e-mt-9af5623f-b68fc\" (UID: \"2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc" Oct 30 18:25:34.841550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.820273 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9af5623f-b68fc\" (UID: \"2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc" Oct 30 18:25:34.841550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.820422 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9af5623f-b68fc\" (UID: \"2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc" Oct 30 18:25:34.929651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.898625 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:25:34.929651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.898716 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:25:35.023085 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.974667 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjqr2\" (UniqueName: \"kubernetes.io/projected/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-kube-api-access-kjqr2\") pod \"maas-api-revoke-keys-e2e-mt-9af5623f-b68fc\" (UID: \"2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc" Oct 30 18:25:35.023085 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.974873 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9af5623f-b68fc\" (UID: \"2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc" Oct 30 18:25:35.023085 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.975103 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9af5623f-b68fc\" (UID: \"2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc" Oct 30 18:25:35.023085 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:34.979304 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-9af5623f-b68fc\" (UID: \"2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc" Oct 30 18:25:35.145531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.134365 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 18:25:35.145531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.134448 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" podUID="7b16e19a-e62c-47b4-b695-ccc7164524b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 18:25:35.183068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.182293 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjqr2\" (UniqueName: \"kubernetes.io/projected/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-kube-api-access-kjqr2\") pod \"maas-api-revoke-keys-e2e-mt-9af5623f-b68fc\" (UID: \"2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc" Oct 30 18:25:35.256938 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.256904 5232 patch_prober.go:29] interesting pod/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 18:25:35.257308 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.257257 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" podUID="961c8dee-4505-4b84-b984-3006b6a95dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 18:25:35.280190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.280150 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a767e3a9-c897-44b6-87c8-b1be86f89ca7" path="/var/lib/kubelet/pods/a767e3a9-c897-44b6-87c8-b1be86f89ca7/volumes" Oct 30 18:25:35.282369 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.282343 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:25:35.307603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.307570 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-9af5623f-b68fc\" (UID: \"2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc" Oct 30 18:25:35.309694 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.308534 5232 patch_prober.go:29] interesting pod/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 18:25:35.337912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.337868 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" podUID="7dbbd730-9357-4f2f-b2d3-f41c877205e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 18:25:35.356037 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.355998 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc" Oct 30 18:25:35.772746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.772646 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:35.791381 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.791344 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:25:35.816637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.816591 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:25:35.829537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.819597 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:35.876683 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.876631 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:35.885991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.885692 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:35.951962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.946903 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 18:25:35.951962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.946975 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" podUID="7b16e19a-e62c-47b4-b695-ccc7164524b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 18:25:35.979548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.978058 5232 patch_prober.go:29] interesting pod/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 18:25:35.979548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.978121 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" podUID="7dbbd730-9357-4f2f-b2d3-f41c877205e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 18:25:36.007532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:35.991561 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:36.070627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.061551 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:36.388790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.384110 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:25:36.388790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.384500 5232 patch_prober.go:29] interesting pod/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 18:25:36.388790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.384568 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" podUID="961c8dee-4505-4b84-b984-3006b6a95dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 18:25:36.622543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.590956 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:25:36.689751 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.680821 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:25:36.740562 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.706913 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:36.740562 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.734621 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw"] Oct 30 18:25:36.808767 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.797751 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:36.808767 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.797873 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:36.836825 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.835650 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:36.867573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.854910 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-fdf4da-6rnvw"] Oct 30 18:25:36.867573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.855007 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:25:36.867573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.855054 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:25:36.867573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.855140 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:25:36.893859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.880756 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc"] Oct 30 18:25:36.893859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.881012 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:25:36.893859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.886851 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:36.893859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.892655 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:25:36.893859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.892712 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:25:37.014548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.993614 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:37.014548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:36.995796 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:25:37.014548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:37.006086 5232 patch_prober.go:29] interesting pod/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 18:25:37.014548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:37.006137 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" podUID="7dbbd730-9357-4f2f-b2d3-f41c877205e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 18:25:37.014548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:37.006653 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 18:25:37.014548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:37.006692 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" podUID="7b16e19a-e62c-47b4-b695-ccc7164524b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 18:25:37.014548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:37.007212 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:25:37.014548 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:25:37.008415 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd0bc3f_424c_4af3_8f57_a7cdc3fc4368.slice/crio-5ce9ced8d33d0a783c4e7c52210a1cec930a1a080ee2bfea8c1a4b6bd2fd18fb WatchSource:0}: Error finding container 5ce9ced8d33d0a783c4e7c52210a1cec930a1a080ee2bfea8c1a4b6bd2fd18fb: Status 404 returned error can't find the container with id 5ce9ced8d33d0a783c4e7c52210a1cec930a1a080ee2bfea8c1a4b6bd2fd18fb Oct 30 18:25:37.049200 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:37.019424 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:25:37.204404 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:37.180398 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:25:37.307725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:37.273950 5232 patch_prober.go:29] interesting pod/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 18:25:37.307725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:37.274006 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" podUID="961c8dee-4505-4b84-b984-3006b6a95dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 18:25:37.307725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:37.302319 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:25:37.398464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:37.398439 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:25:37.522751 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:37.522723 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:25:37.721916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:37.719226 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": context deadline exceeded" Oct 30 18:25:37.962350 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:37.962306 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 18:25:38.202912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:38.201665 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" podUID="7b16e19a-e62c-47b4-b695-ccc7164524b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 18:25:38.203402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:38.134296 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:25:38.205745 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:38.007606 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:25:38.206159 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:37.974012 5232 patch_prober.go:29] interesting pod/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 18:25:38.206451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:38.206416 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" podUID="7dbbd730-9357-4f2f-b2d3-f41c877205e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 18:25:38.246831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:38.187599 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:25:38.310529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:38.304321 5232 patch_prober.go:29] interesting pod/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 18:25:38.310529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:38.304392 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" podUID="961c8dee-4505-4b84-b984-3006b6a95dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 18:25:38.590357 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:38.590319 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc" event={"ID":"2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368","Type":"ContainerStarted","Data":"5ce9ced8d33d0a783c4e7c52210a1cec930a1a080ee2bfea8c1a4b6bd2fd18fb"} Oct 30 18:25:38.850588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:38.846164 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e212d8b5-6490-48e7-b2f4-636d017e9b29" path="/var/lib/kubelet/pods/e212d8b5-6490-48e7-b2f4-636d017e9b29/volumes" Oct 30 18:25:38.965538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:38.962570 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 18:25:38.965538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:38.962636 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" podUID="7b16e19a-e62c-47b4-b695-ccc7164524b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 18:25:38.979534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:38.975859 5232 patch_prober.go:29] interesting pod/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" start-of-body= Oct 30 18:25:38.979534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:38.975917 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" podUID="7dbbd730-9357-4f2f-b2d3-f41c877205e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.169:15021/healthz/ready\": dial tcp 10.133.0.169:15021: connect: connection refused" Oct 30 18:25:39.095684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.095624 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4"] Oct 30 18:25:39.103549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.103435 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4"] Oct 30 18:25:39.104575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.103981 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" Oct 30 18:25:39.163652 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.135873 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-227c01-serving-cert\"" Oct 30 18:25:39.205086 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.205040 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-maas-api-tls\") pod \"maas-api-e2e-api-b-227c01-5794c658f5-l8gx4\" (UID: \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" Oct 30 18:25:39.205323 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.205157 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gmj8m\" (UniqueName: \"kubernetes.io/projected/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-kube-api-access-gmj8m\") pod \"maas-api-e2e-api-b-227c01-5794c658f5-l8gx4\" (UID: \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" Oct 30 18:25:39.205417 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.205392 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-openshift-service-ca\") pod \"maas-api-e2e-api-b-227c01-5794c658f5-l8gx4\" (UID: \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" Oct 30 18:25:39.205802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.205770 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-metrics-certs\") pod \"maas-api-e2e-api-b-227c01-5794c658f5-l8gx4\" (UID: \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" Oct 30 18:25:39.245941 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.245898 5232 patch_prober.go:29] interesting pod/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 18:25:39.246224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.246191 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" podUID="961c8dee-4505-4b84-b984-3006b6a95dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 18:25:39.319532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.317206 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-metrics-certs\") pod \"maas-api-e2e-api-b-227c01-5794c658f5-l8gx4\" (UID: \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" Oct 30 18:25:39.319532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.317286 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-maas-api-tls\") pod \"maas-api-e2e-api-b-227c01-5794c658f5-l8gx4\" (UID: \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" Oct 30 18:25:39.319532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.317363 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gmj8m\" (UniqueName: \"kubernetes.io/projected/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-kube-api-access-gmj8m\") pod \"maas-api-e2e-api-b-227c01-5794c658f5-l8gx4\" (UID: \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" Oct 30 18:25:39.319532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.317635 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-openshift-service-ca\") pod \"maas-api-e2e-api-b-227c01-5794c658f5-l8gx4\" (UID: \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" Oct 30 18:25:39.319532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.318494 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-openshift-service-ca\") pod \"maas-api-e2e-api-b-227c01-5794c658f5-l8gx4\" (UID: \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" Oct 30 18:25:39.403590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.402130 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-maas-api-tls\") pod \"maas-api-e2e-api-b-227c01-5794c658f5-l8gx4\" (UID: \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" Oct 30 18:25:39.481536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.472096 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-metrics-certs\") pod \"maas-api-e2e-api-b-227c01-5794c658f5-l8gx4\" (UID: \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" Oct 30 18:25:39.481536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.472660 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gmj8m\" (UniqueName: \"kubernetes.io/projected/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-kube-api-access-gmj8m\") pod \"maas-api-e2e-api-b-227c01-5794c658f5-l8gx4\" (UID: \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" Oct 30 18:25:39.501534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.489961 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" Oct 30 18:25:39.569689 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:25:39.549065 5232 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cd0bc3f_424c_4af3_8f57_a7cdc3fc4368.slice/crio-conmon-38ec60b5bfdba81bcaf0808cae8919650875c1d74c3ce3863a8218858f60d018.scope\": RecentStats: unable to find data in memory cache]" Oct 30 18:25:39.682944 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.682908 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:25:39.691277 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.691201 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4"] Oct 30 18:25:39.692302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.691429 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" podUID="29a994e2-22c0-49ce-8d6b-1898bf2f7edc" containerName="discovery" containerID="cri-o://e8879e631e66c9aec5f31638b7c251c79ab3bdeff767bce50b0940038e90ee0c" gracePeriod=30 Oct 30 18:25:39.701654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.701396 5232 generic.go:356] "Generic (PLEG): container finished" podID="2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368" containerID="38ec60b5bfdba81bcaf0808cae8919650875c1d74c3ce3863a8218858f60d018" exitCode=0 Oct 30 18:25:39.701654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.701470 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc" event={"ID":"2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368","Type":"ContainerDied","Data":"38ec60b5bfdba81bcaf0808cae8919650875c1d74c3ce3863a8218858f60d018"} Oct 30 18:25:39.963059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.948967 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" start-of-body= Oct 30 18:25:39.963059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.949027 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" podUID="7b16e19a-e62c-47b4-b695-ccc7164524b5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.172:15021/healthz/ready\": dial tcp 10.133.0.172:15021: connect: connection refused" Oct 30 18:25:39.997869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:39.997822 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:25:40.001106 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.001081 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:25:40.002339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.002275 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:25:40.345033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.343871 5232 patch_prober.go:29] interesting pod/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" start-of-body= Oct 30 18:25:40.345033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.343931 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" podUID="961c8dee-4505-4b84-b984-3006b6a95dd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.155:15021/healthz/ready\": dial tcp 10.133.0.155:15021: connect: connection refused" Oct 30 18:25:40.396283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.396255 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4"] Oct 30 18:25:40.586438 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.586398 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:25:40.683716 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.680572 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-csr-ca-configmap\") pod \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " Oct 30 18:25:40.683716 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.680770 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-token\" (UniqueName: \"kubernetes.io/projected/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-token\") pod \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " Oct 30 18:25:40.683716 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.680970 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-kube-api-access-fr8w7\" (UniqueName: \"kubernetes.io/projected/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-kube-api-access-fr8w7\") pod \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " Oct 30 18:25:40.683716 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.681040 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-cacerts\" (UniqueName: \"kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-cacerts\") pod \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " Oct 30 18:25:40.683716 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.681087 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-local-certs\") pod \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " Oct 30 18:25:40.683716 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.681119 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-csr-dns-cert\") pod \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " Oct 30 18:25:40.683716 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.681187 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-kubeconfig\") pod \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\" (UID: \"29a994e2-22c0-49ce-8d6b-1898bf2f7edc\") " Oct 30 18:25:40.702851 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.702812 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-csr-ca-configmap" pod "29a994e2-22c0-49ce-8d6b-1898bf2f7edc" (UID: "29a994e2-22c0-49ce-8d6b-1898bf2f7edc"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:40.889960 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.889813 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-token" pod "29a994e2-22c0-49ce-8d6b-1898bf2f7edc" (UID: "29a994e2-22c0-49ce-8d6b-1898bf2f7edc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:40.902796 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.900769 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-cacerts" pod "29a994e2-22c0-49ce-8d6b-1898bf2f7edc" (UID: "29a994e2-22c0-49ce-8d6b-1898bf2f7edc"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:40.902796 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.901159 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-local-certs" pod "29a994e2-22c0-49ce-8d6b-1898bf2f7edc" (UID: "29a994e2-22c0-49ce-8d6b-1898bf2f7edc"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:25:40.902796 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.901385 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-kubeconfig" pod "29a994e2-22c0-49ce-8d6b-1898bf2f7edc" (UID: "29a994e2-22c0-49ce-8d6b-1898bf2f7edc"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:40.939585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.930065 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-csr-dns-cert" pod "29a994e2-22c0-49ce-8d6b-1898bf2f7edc" (UID: "29a994e2-22c0-49ce-8d6b-1898bf2f7edc"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:40.939585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:40.930231 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-kube-api-access-fr8w7" pod "29a994e2-22c0-49ce-8d6b-1898bf2f7edc" (UID: "29a994e2-22c0-49ce-8d6b-1898bf2f7edc"). InnerVolumeSpecName "kube-api-access-fr8w7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:41.018763 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.018727 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" event={"ID":"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10","Type":"ContainerStarted","Data":"35cdf398d2d166ef6c28e6433d019d095528a26bbc09fc8d72bd8d76faea889f"} Oct 30 18:25:41.035618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.035589 5232 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-cacerts\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:41.035776 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.035754 5232 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-local-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:41.035888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.035876 5232 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-csr-dns-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:41.036005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.035992 5232 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-kubeconfig\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:41.036098 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.036086 5232 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-csr-ca-configmap\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:41.036187 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.036176 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:41.036302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.036291 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fr8w7\" (UniqueName: \"kubernetes.io/projected/29a994e2-22c0-49ce-8d6b-1898bf2f7edc-kube-api-access-fr8w7\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:41.036495 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.036461 5232 generic.go:356] "Generic (PLEG): container finished" podID="29a994e2-22c0-49ce-8d6b-1898bf2f7edc" containerID="e8879e631e66c9aec5f31638b7c251c79ab3bdeff767bce50b0940038e90ee0c" exitCode=0 Oct 30 18:25:41.036904 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.036881 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" Oct 30 18:25:41.036997 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.036944 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" event={"ID":"29a994e2-22c0-49ce-8d6b-1898bf2f7edc","Type":"ContainerDied","Data":"e8879e631e66c9aec5f31638b7c251c79ab3bdeff767bce50b0940038e90ee0c"} Oct 30 18:25:41.036997 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.036992 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4" event={"ID":"29a994e2-22c0-49ce-8d6b-1898bf2f7edc","Type":"ContainerDied","Data":"7000becc2b9b477c577416069f5745690fc85473facedff80bb31d387a8d8ee1"} Oct 30 18:25:41.037111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.037035 5232 scope.go:122] "RemoveContainer" containerID="e8879e631e66c9aec5f31638b7c251c79ab3bdeff767bce50b0940038e90ee0c" Oct 30 18:25:41.071261 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.071234 5232 scope.go:122] "RemoveContainer" containerID="e8879e631e66c9aec5f31638b7c251c79ab3bdeff767bce50b0940038e90ee0c" Oct 30 18:25:41.071717 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:25:41.071692 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e8879e631e66c9aec5f31638b7c251c79ab3bdeff767bce50b0940038e90ee0c\": container with ID starting with e8879e631e66c9aec5f31638b7c251c79ab3bdeff767bce50b0940038e90ee0c not found: ID does not exist" containerID="e8879e631e66c9aec5f31638b7c251c79ab3bdeff767bce50b0940038e90ee0c" Oct 30 18:25:41.071808 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.071732 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e8879e631e66c9aec5f31638b7c251c79ab3bdeff767bce50b0940038e90ee0c"} err="failed to get container status \"e8879e631e66c9aec5f31638b7c251c79ab3bdeff767bce50b0940038e90ee0c\": rpc error: code = NotFound desc = could not find container \"e8879e631e66c9aec5f31638b7c251c79ab3bdeff767bce50b0940038e90ee0c\": container with ID starting with e8879e631e66c9aec5f31638b7c251c79ab3bdeff767bce50b0940038e90ee0c not found: ID does not exist" Oct 30 18:25:41.098184 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.098142 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4"] Oct 30 18:25:41.104261 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.104229 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-2s5j4"] Oct 30 18:25:41.186630 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.185006 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:25:41.187209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.187184 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:25:41.251117 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.251089 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:25:41.253418 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.252828 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:25:41.368307 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.368272 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc" Oct 30 18:25:41.469082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.469052 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-kube-api-access-kjqr2\" (UniqueName: \"kubernetes.io/projected/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-kube-api-access-kjqr2\") pod \"2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368\" (UID: \"2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368\") " Oct 30 18:25:41.469277 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.469206 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-maas-api-openshift-ca\") pod \"2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368\" (UID: \"2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368\") " Oct 30 18:25:41.469379 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.469359 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-maas-api-xks-ca\") pod \"2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368\" (UID: \"2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368\") " Oct 30 18:25:41.469720 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.469689 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-maas-api-openshift-ca" pod "2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368" (UID: "2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:25:41.470918 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.470890 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:41.471944 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.471911 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-kube-api-access-kjqr2" pod "2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368" (UID: "2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368"). InnerVolumeSpecName "kube-api-access-kjqr2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:25:41.472295 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.472272 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-maas-api-xks-ca" pod "2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368" (UID: "2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:25:41.572150 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.572118 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjqr2\" (UniqueName: \"kubernetes.io/projected/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-kube-api-access-kjqr2\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:41.572150 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.572147 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:25:41.932842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.932808 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh" Oct 30 18:25:41.933255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:41.932882 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb" Oct 30 18:25:42.052530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:42.050524 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc" Oct 30 18:25:42.052530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:42.050577 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc" event={"ID":"2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368","Type":"ContainerDied","Data":"5ce9ced8d33d0a783c4e7c52210a1cec930a1a080ee2bfea8c1a4b6bd2fd18fb"} Oct 30 18:25:42.052530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:42.050648 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5ce9ced8d33d0a783c4e7c52210a1cec930a1a080ee2bfea8c1a4b6bd2fd18fb" Oct 30 18:25:42.053219 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:42.053191 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" event={"ID":"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10","Type":"ContainerStarted","Data":"29faeca69bacad3df2b82e9587c9805604d3b8d39ce4e1689d854017f6c3a01a"} Oct 30 18:25:42.053986 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:42.053967 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" Oct 30 18:25:42.106566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:42.106492 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" podStartSLOduration=3.432859207 podStartE2EDuration="4.106477084s" totalImagesPullingTime="673.617877ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:25:42.103680205 +0000 UTC m=+1441.420100527" watchObservedRunningTime="2026-10-30 18:25:42.106477084 +0000 UTC m=+1441.422897404" Oct 30 18:25:42.806082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:42.806050 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29a994e2-22c0-49ce-8d6b-1898bf2f7edc" path="/var/lib/kubelet/pods/29a994e2-22c0-49ce-8d6b-1898bf2f7edc/volumes" Oct 30 18:25:47.498533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:47.497793 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:25:47.539611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:47.539322 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": context deadline exceeded" Oct 30 18:25:47.570667 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:47.570615 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:47.617651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:47.594691 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:47.617985 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:47.615249 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:25:48.019533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:48.018619 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:48.052277 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:48.052145 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:48.082332 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:48.081204 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:25:48.147533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:48.124877 5232 scope.go:122] "RemoveContainer" containerID="c17daf60b29c766d686e09128df2187bd33b5bfabd572fe4c08b967e9298772a" Oct 30 18:25:48.147945 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:48.147912 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" Oct 30 18:25:48.289610 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:48.263745 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": context deadline exceeded" Oct 30 18:25:48.326140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:48.291327 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:25:48.351846 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:48.351788 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:48.569775 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:48.569675 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:48.618723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:48.618672 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:48.619394 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:48.619349 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:25:48.620228 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:48.620207 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:25:48.620446 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:48.620431 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:25:48.709599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:48.673701 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:48.709599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:48.674257 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:25:48.723586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:48.699316 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:25:48.824054 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:48.823937 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": context deadline exceeded" Oct 30 18:25:49.114597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.110663 5232 scope.go:122] "RemoveContainer" containerID="04b3ad726c57bc91c23613c8acad7eb23abb5393d67149e878006af67df5d815" Oct 30 18:25:49.123591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.122658 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:49.319561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.294330 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:25:49.343539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.330710 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:25:49.343539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.331206 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:25:49.343539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.331538 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:25:49.468272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.468116 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:49.514533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.506037 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:25:49.514533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.506272 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:25:49.514533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.506341 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:25:49.514533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.506400 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:25:49.635686 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.635640 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:49.671015 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.661801 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:49.722597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.703672 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:49.722597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.709726 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:25:49.722597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.710146 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": context deadline exceeded" Oct 30 18:25:49.729540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.729332 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:25:49.729540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.729443 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:25:49.740943 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.740893 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:49.758425 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.757297 5232 scope.go:122] "RemoveContainer" containerID="6a1a8c80a4f4e9ea5a7b6234181db2931b6b2977b6f5c5ec6a3a16faa63e7bca" Oct 30 18:25:49.770622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.759031 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:25:49.959672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:49.959591 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:50.131849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:50.129439 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2" podUID="693cdf78-86cc-4f53-9c17-3926b662aeb4" containerName="main" probeResult="failure" output="Get \"https://10.133.0.26:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:50.131849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:50.129815 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:50.131849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:50.130018 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:50.297892 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:50.297843 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:50.332698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:50.332638 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:50.333031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:50.333006 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:25:50.333351 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:50.333333 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:25:50.333568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:50.333276 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:25:50.334045 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:50.334030 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:25:50.572537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:50.551396 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded" Oct 30 18:25:50.608591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:50.606978 5232 scope.go:122] "RemoveContainer" containerID="03f0b05d7e72dbc45e263c587fb1bba281dac4b35395432219db05780e4b0b90" Oct 30 18:25:50.675789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:50.675686 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:25:50.676978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:50.676949 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 18:25:50.735638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:50.735587 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:25:50.746538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:50.740195 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:50.783335 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:50.783308 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:25:50.791311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:50.791273 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:51.135557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:51.124816 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:25:51.136084 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:51.135845 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:51.310773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:51.310661 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:25:51.589374 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:51.589269 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:51.596819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:51.596777 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:25:51.597102 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:51.597074 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:25:51.639452 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:51.639192 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:51.745380 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:51.732651 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:51.893632 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:51.875725 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:51.893632 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:51.878633 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:51.893632 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:51.879918 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:52.131574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:52.126090 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:25:52.159778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:52.159718 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:52.377549 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:25:52.370170 5232 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a994e2_22c0_49ce_8d6b_1898bf2f7edc.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:25:52.599700 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:52.599653 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:25:52.600373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:52.599928 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:25:52.625536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:52.599972 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:25:52.752263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:52.752212 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:52.753285 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:52.753253 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:25:52.823728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:52.823691 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:25:52.886708 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:52.886646 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:52.953565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:52.908927 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:52.953565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:52.909389 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 18:25:53.131633 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:53.131579 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:53.164177 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:53.164128 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:25:53.377788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:53.377614 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc"] Oct 30 18:25:53.627733 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:53.627661 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:25:53.636581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:53.628125 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:25:53.718790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:53.698340 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:53.770102 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:53.770064 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": context deadline exceeded" Oct 30 18:25:53.771567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:53.771543 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:25:53.793537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:53.773594 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:25:53.834810 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:53.834749 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:53.837624 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:53.837585 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:25:53.924377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:53.924334 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:53.925142 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:53.925085 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:53.936584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:53.932809 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:25:54.163610 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:54.143383 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:54.203746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:54.179739 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:54.398558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:54.372802 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:54.543926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:54.543881 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-9af5623f-b68fc"] Oct 30 18:25:54.631533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:54.629066 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:25:54.711634 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:54.711592 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:54.717593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:54.707819 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:25:54.772177 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:54.771787 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:54.831137 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:54.831034 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:54.853663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:54.853604 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:54.913536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:54.910537 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368" path="/var/lib/kubelet/pods/2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368/volumes" Oct 30 18:25:54.934532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:54.925748 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:54.936626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:54.936588 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:54.944642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:54.944604 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:55.170658 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.170563 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm"] Oct 30 18:25:55.172052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.172021 5232 cpu_manager.go:414] "containerMap: removing container" podUID="2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368" containerName="revoke-keys" Oct 30 18:25:55.173134 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.173115 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368" containerName="revoke-keys" Oct 30 18:25:55.454983 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.454888 5232 cpu_manager.go:414] "containerMap: removing container" podUID="29a994e2-22c0-49ce-8d6b-1898bf2f7edc" containerName="discovery" Oct 30 18:25:55.455188 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.455170 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29a994e2-22c0-49ce-8d6b-1898bf2f7edc" containerName="discovery" Oct 30 18:25:55.455671 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.455650 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="29a994e2-22c0-49ce-8d6b-1898bf2f7edc" containerName="discovery" Oct 30 18:25:55.457075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.457057 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cd0bc3f-424c-4af3-8f57-a7cdc3fc4368" containerName="revoke-keys" Oct 30 18:25:55.471775 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.333811 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:55.487536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.333765 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:55.488285 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.475610 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm"] Oct 30 18:25:55.488483 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.475697 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm" Oct 30 18:25:55.518959 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:25:55.505344 5232 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a994e2_22c0_49ce_8d6b_1898bf2f7edc.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:25:55.518959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.514099 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:25:55.543703 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.543667 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:25:55.605548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.598496 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9da94e-bcbf-4490-89f4-cce4c3505f88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-295497-h8kfm\" (UID: \"1c9da94e-bcbf-4490-89f4-cce4c3505f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm" Oct 30 18:25:55.605548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.598587 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9da94e-bcbf-4490-89f4-cce4c3505f88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-295497-h8kfm\" (UID: \"1c9da94e-bcbf-4490-89f4-cce4c3505f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm" Oct 30 18:25:55.605548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.598671 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z49bk\" (UniqueName: \"kubernetes.io/projected/1c9da94e-bcbf-4490-89f4-cce4c3505f88-kube-api-access-z49bk\") pod \"maas-api-revoke-keys-e2e-api-a-295497-h8kfm\" (UID: \"1c9da94e-bcbf-4490-89f4-cce4c3505f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm" Oct 30 18:25:55.686593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.658856 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:25:55.738596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.712949 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:55.738596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.718035 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:55.738596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.726736 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9da94e-bcbf-4490-89f4-cce4c3505f88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-295497-h8kfm\" (UID: \"1c9da94e-bcbf-4490-89f4-cce4c3505f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm" Oct 30 18:25:55.738596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.726905 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z49bk\" (UniqueName: \"kubernetes.io/projected/1c9da94e-bcbf-4490-89f4-cce4c3505f88-kube-api-access-z49bk\") pod \"maas-api-revoke-keys-e2e-api-a-295497-h8kfm\" (UID: \"1c9da94e-bcbf-4490-89f4-cce4c3505f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm" Oct 30 18:25:55.738596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.727423 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9da94e-bcbf-4490-89f4-cce4c3505f88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-295497-h8kfm\" (UID: \"1c9da94e-bcbf-4490-89f4-cce4c3505f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm" Oct 30 18:25:55.745684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.745645 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9da94e-bcbf-4490-89f4-cce4c3505f88-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-295497-h8kfm\" (UID: \"1c9da94e-bcbf-4490-89f4-cce4c3505f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm" Oct 30 18:25:55.789046 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.789005 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:55.814574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.798692 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z49bk\" (UniqueName: \"kubernetes.io/projected/1c9da94e-bcbf-4490-89f4-cce4c3505f88-kube-api-access-z49bk\") pod \"maas-api-revoke-keys-e2e-api-a-295497-h8kfm\" (UID: \"1c9da94e-bcbf-4490-89f4-cce4c3505f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm" Oct 30 18:25:55.814574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.799048 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9da94e-bcbf-4490-89f4-cce4c3505f88-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-295497-h8kfm\" (UID: \"1c9da94e-bcbf-4490-89f4-cce4c3505f88\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm" Oct 30 18:25:55.833933 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.833891 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:55.842211 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.842184 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:25:55.854231 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.854192 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:55.927756 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.926607 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:55.976730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.976680 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:25:56.005592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:55.988605 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:25:56.051561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:56.043923 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm" Oct 30 18:25:56.477140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:56.477093 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:25:56.501886 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:56.501840 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:56.669605 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:56.669440 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:56.735534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:56.734995 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:56.794536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:56.790706 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:56.878957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:56.876388 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:56.878957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:56.876958 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:25:56.997486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:56.965864 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 18:25:56.997486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:56.984691 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:25:57.032537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:57.006865 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:57.479535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:57.478712 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:57.483532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:57.481906 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:57.512536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:57.511760 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:57.625626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:57.571780 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:57.717537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:57.631707 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:25:57.717537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:57.684849 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:57.743161 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:57.741195 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:57.796320 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:57.796277 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:57.912711 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:57.885433 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:57.912711 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:57.885824 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:57.912711 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:57.893716 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm"] Oct 30 18:25:57.933793 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:25:57.933724 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c9da94e_bcbf_4490_89f4_cce4c3505f88.slice/crio-cc37c46e3bbc1294554fdb49002ae315689472a6747ff56bef824af3077358f2 WatchSource:0}: Error finding container cc37c46e3bbc1294554fdb49002ae315689472a6747ff56bef824af3077358f2: Status 404 returned error can't find the container with id cc37c46e3bbc1294554fdb49002ae315689472a6747ff56bef824af3077358f2 Oct 30 18:25:57.984538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:57.966132 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": context deadline exceeded" Oct 30 18:25:58.016418 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:57.985935 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:25:58.174703 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.173700 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:25:58.206360 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.195287 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/health\": context deadline exceeded" Oct 30 18:25:58.499612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.497430 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:58.583846 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.583039 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n"] Oct 30 18:25:58.584241 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.584219 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n"] Oct 30 18:25:58.584718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.584687 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" containerID="cri-o://e0aa7425dbd1d5ab06058423414ddc13e5760d0762cf4367ec7a4288c5666d94" gracePeriod=30 Oct 30 18:25:58.586787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.586754 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:58.608565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.594617 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9"] Oct 30 18:25:58.608565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.594912 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" containerID="cri-o://7ef1400449bd017435f6b7ce923a64cb64a6145efafc6015ec46f4e75e959766" gracePeriod=30 Oct 30 18:25:58.608565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.595211 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" podUID="bf05a858-37f7-4ce6-a4ba-5431a372d1b3" containerName="istio-proxy" containerID="cri-o://d80b9482c4286db5bd9b75ee0ce8bd8f826ff6b8d324580e6a2f7d12dcd678fb" gracePeriod=30 Oct 30 18:25:58.623971 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.620583 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s"] Oct 30 18:25:58.623971 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.621074 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" podUID="d1fd255d-3e8c-489a-9151-cb9533af2405" containerName="istio-proxy" containerID="cri-o://2c34e13eac894b19ed5ae92d3d22c57afd4637ac1255e4a25e2dccc5862ad943" gracePeriod=30 Oct 30 18:25:58.623971 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:25:58.622206 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:25:58.686557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.643004 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:58.686557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.643450 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:25:58.686557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.679597 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg"] Oct 30 18:25:58.713535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.687224 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" podUID="352fefd4-5ce5-4ce2-81ff-28769154e70b" containerName="istio-proxy" containerID="cri-o://e9ed2e2dea302a6b56108124bc5cc3debc43fe6a6f0700f69de5c79b309b010d" gracePeriod=30 Oct 30 18:25:58.713535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.704722 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:25:58.713535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.705719 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:25:58.766898 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.763475 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:58.770131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.770083 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm" event={"ID":"1c9da94e-bcbf-4490-89f4-cce4c3505f88","Type":"ContainerStarted","Data":"cc37c46e3bbc1294554fdb49002ae315689472a6747ff56bef824af3077358f2"} Oct 30 18:25:58.810526 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.810467 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:58.903529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.895257 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:25:58.903529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:58.895698 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:25:59.000312 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:59.000270 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:59.033162 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:59.025183 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:25:59.193541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:59.192612 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:59.378531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:59.365767 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": context deadline exceeded" Oct 30 18:25:59.405547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:59.404117 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:59.499695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:59.499236 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:25:59.502974 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:59.502443 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:25:59.504733 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:59.504702 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:25:59.506071 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:59.506009 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:59.516656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:59.508720 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:25:59.568117 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:59.566139 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:25:59.604558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:59.600128 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:25:59.644100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:59.643604 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:25:59.688756 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:59.687973 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:25:59.703885 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:59.703861 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:25:59.806815 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:59.791393 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:25:59.806815 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:59.802242 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:25:59.880968 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:25:59.880844 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm" event={"ID":"1c9da94e-bcbf-4490-89f4-cce4c3505f88","Type":"ContainerStarted","Data":"b5fd730e9b318de0b4e91bc809bb38ce3b0030f778059f6a095f198f7c7ca6e9"} Oct 30 18:26:00.151349 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:00.151238 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm" podStartSLOduration=5.151216278 podStartE2EDuration="5.151216278s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:00.144939286 +0000 UTC m=+1459.461359608" watchObservedRunningTime="2026-10-30 18:26:00.151216278 +0000 UTC m=+1459.467636599" Oct 30 18:26:00.895980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:00.895942 5232 generic.go:356] "Generic (PLEG): container finished" podID="1c9da94e-bcbf-4490-89f4-cce4c3505f88" containerID="b5fd730e9b318de0b4e91bc809bb38ce3b0030f778059f6a095f198f7c7ca6e9" exitCode=0 Oct 30 18:26:00.896454 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:00.896090 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm" event={"ID":"1c9da94e-bcbf-4490-89f4-cce4c3505f88","Type":"ContainerDied","Data":"b5fd730e9b318de0b4e91bc809bb38ce3b0030f778059f6a095f198f7c7ca6e9"} Oct 30 18:26:01.803092 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:01.803034 5232 patch_prober.go:29] interesting pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:26:01.803271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:01.803099 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" podUID="43d92d4d-f62f-4b68-a9bf-89186482a244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:02.313567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.313264 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm" Oct 30 18:26:02.348155 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.348089 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c9da94e-bcbf-4490-89f4-cce4c3505f88-kube-api-access-z49bk\" (UniqueName: \"kubernetes.io/projected/1c9da94e-bcbf-4490-89f4-cce4c3505f88-kube-api-access-z49bk\") pod \"1c9da94e-bcbf-4490-89f4-cce4c3505f88\" (UID: \"1c9da94e-bcbf-4490-89f4-cce4c3505f88\") " Oct 30 18:26:02.348340 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.348223 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c9da94e-bcbf-4490-89f4-cce4c3505f88-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9da94e-bcbf-4490-89f4-cce4c3505f88-maas-api-openshift-ca\") pod \"1c9da94e-bcbf-4490-89f4-cce4c3505f88\" (UID: \"1c9da94e-bcbf-4490-89f4-cce4c3505f88\") " Oct 30 18:26:02.348947 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.348916 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1c9da94e-bcbf-4490-89f4-cce4c3505f88-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9da94e-bcbf-4490-89f4-cce4c3505f88-maas-api-xks-ca\") pod \"1c9da94e-bcbf-4490-89f4-cce4c3505f88\" (UID: \"1c9da94e-bcbf-4490-89f4-cce4c3505f88\") " Oct 30 18:26:02.349953 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.349902 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c9da94e-bcbf-4490-89f4-cce4c3505f88-maas-api-openshift-ca" pod "1c9da94e-bcbf-4490-89f4-cce4c3505f88" (UID: "1c9da94e-bcbf-4490-89f4-cce4c3505f88"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:02.350383 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.350341 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1c9da94e-bcbf-4490-89f4-cce4c3505f88-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:02.354016 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.353991 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c9da94e-bcbf-4490-89f4-cce4c3505f88-kube-api-access-z49bk" pod "1c9da94e-bcbf-4490-89f4-cce4c3505f88" (UID: "1c9da94e-bcbf-4490-89f4-cce4c3505f88"). InnerVolumeSpecName "kube-api-access-z49bk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:02.361375 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.361338 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1c9da94e-bcbf-4490-89f4-cce4c3505f88-maas-api-xks-ca" pod "1c9da94e-bcbf-4490-89f4-cce4c3505f88" (UID: "1c9da94e-bcbf-4490-89f4-cce4c3505f88"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:02.453000 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.452914 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1c9da94e-bcbf-4490-89f4-cce4c3505f88-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:02.453000 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.452958 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z49bk\" (UniqueName: \"kubernetes.io/projected/1c9da94e-bcbf-4490-89f4-cce4c3505f88-kube-api-access-z49bk\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:02.536636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.536047 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x"] Oct 30 18:26:02.539601 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.537775 5232 cpu_manager.go:414] "containerMap: removing container" podUID="1c9da94e-bcbf-4490-89f4-cce4c3505f88" containerName="revoke-keys" Oct 30 18:26:02.539601 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.537802 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c9da94e-bcbf-4490-89f4-cce4c3505f88" containerName="revoke-keys" Oct 30 18:26:02.539601 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.537979 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c9da94e-bcbf-4490-89f4-cce4c3505f88" containerName="revoke-keys" Oct 30 18:26:02.543109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.542289 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x" Oct 30 18:26:02.571331 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.571223 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/598f6936-6be8-480f-b11f-62fc8bc62c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6c4a3a5-89j8x\" (UID: \"598f6936-6be8-480f-b11f-62fc8bc62c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x" Oct 30 18:26:02.572123 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.571680 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n75kt\" (UniqueName: \"kubernetes.io/projected/598f6936-6be8-480f-b11f-62fc8bc62c07-kube-api-access-n75kt\") pod \"maas-api-revoke-sub-c6c4a3a5-89j8x\" (UID: \"598f6936-6be8-480f-b11f-62fc8bc62c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x" Oct 30 18:26:02.572123 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.571732 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/598f6936-6be8-480f-b11f-62fc8bc62c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6c4a3a5-89j8x\" (UID: \"598f6936-6be8-480f-b11f-62fc8bc62c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x" Oct 30 18:26:02.599367 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.599336 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x"] Oct 30 18:26:02.620199 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:02.620162 5232 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a994e2_22c0_49ce_8d6b_1898bf2f7edc.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:26:02.679529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.679473 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/598f6936-6be8-480f-b11f-62fc8bc62c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6c4a3a5-89j8x\" (UID: \"598f6936-6be8-480f-b11f-62fc8bc62c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x" Oct 30 18:26:02.679855 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.679829 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n75kt\" (UniqueName: \"kubernetes.io/projected/598f6936-6be8-480f-b11f-62fc8bc62c07-kube-api-access-n75kt\") pod \"maas-api-revoke-sub-c6c4a3a5-89j8x\" (UID: \"598f6936-6be8-480f-b11f-62fc8bc62c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x" Oct 30 18:26:02.680028 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.680010 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/598f6936-6be8-480f-b11f-62fc8bc62c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6c4a3a5-89j8x\" (UID: \"598f6936-6be8-480f-b11f-62fc8bc62c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x" Oct 30 18:26:02.681563 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.681534 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/598f6936-6be8-480f-b11f-62fc8bc62c07-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c6c4a3a5-89j8x\" (UID: \"598f6936-6be8-480f-b11f-62fc8bc62c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x" Oct 30 18:26:02.690840 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.684018 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/598f6936-6be8-480f-b11f-62fc8bc62c07-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c6c4a3a5-89j8x\" (UID: \"598f6936-6be8-480f-b11f-62fc8bc62c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x" Oct 30 18:26:02.712054 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.712009 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n75kt\" (UniqueName: \"kubernetes.io/projected/598f6936-6be8-480f-b11f-62fc8bc62c07-kube-api-access-n75kt\") pod \"maas-api-revoke-sub-c6c4a3a5-89j8x\" (UID: \"598f6936-6be8-480f-b11f-62fc8bc62c07\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x" Oct 30 18:26:02.898332 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.898292 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x" Oct 30 18:26:02.939707 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.939662 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm" event={"ID":"1c9da94e-bcbf-4490-89f4-cce4c3505f88","Type":"ContainerDied","Data":"cc37c46e3bbc1294554fdb49002ae315689472a6747ff56bef824af3077358f2"} Oct 30 18:26:02.939899 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.939736 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc37c46e3bbc1294554fdb49002ae315689472a6747ff56bef824af3077358f2" Oct 30 18:26:02.939899 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:02.939842 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm" Oct 30 18:26:03.258114 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:26:03.258083 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod598f6936_6be8_480f_b11f_62fc8bc62c07.slice/crio-0fcb8416c4c91aed2371775f1e8d0f3a3ff2340ab2c770d3b27cf8d894054072 WatchSource:0}: Error finding container 0fcb8416c4c91aed2371775f1e8d0f3a3ff2340ab2c770d3b27cf8d894054072: Status 404 returned error can't find the container with id 0fcb8416c4c91aed2371775f1e8d0f3a3ff2340ab2c770d3b27cf8d894054072 Oct 30 18:26:03.260359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:03.260324 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x"] Oct 30 18:26:03.954938 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:03.954897 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x" event={"ID":"598f6936-6be8-480f-b11f-62fc8bc62c07","Type":"ContainerStarted","Data":"97ae47a976812e07a85a90f9e7d486e51dd7159689a4faffce6ec3c82632e0f6"} Oct 30 18:26:03.955420 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:03.954966 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x" event={"ID":"598f6936-6be8-480f-b11f-62fc8bc62c07","Type":"ContainerStarted","Data":"0fcb8416c4c91aed2371775f1e8d0f3a3ff2340ab2c770d3b27cf8d894054072"} Oct 30 18:26:04.016787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:04.016730 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x" podStartSLOduration=2.016715449 podStartE2EDuration="2.016715449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:04.013074827 +0000 UTC m=+1463.329495148" watchObservedRunningTime="2026-10-30 18:26:04.016715449 +0000 UTC m=+1463.333135770" Oct 30 18:26:04.127587 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:04.123795 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7274-90e33e"] Oct 30 18:26:04.243184 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:04.242867 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7274-90e33e"] Oct 30 18:26:04.243184 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:04.242991 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7274-90e33e" Oct 30 18:26:04.258599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:04.258569 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-n7rzn\"" Oct 30 18:26:04.316983 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:04.316944 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74525\" (UniqueName: \"kubernetes.io/projected/ae3b1f01-05b6-469f-9966-1336b599f53e-kube-api-access-74525\") pod \"test-curl-7274-90e33e\" (UID: \"ae3b1f01-05b6-469f-9966-1336b599f53e\") " pod="opendatahub/test-curl-7274-90e33e" Oct 30 18:26:04.419140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:04.419100 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74525\" (UniqueName: \"kubernetes.io/projected/ae3b1f01-05b6-469f-9966-1336b599f53e-kube-api-access-74525\") pod \"test-curl-7274-90e33e\" (UID: \"ae3b1f01-05b6-469f-9966-1336b599f53e\") " pod="opendatahub/test-curl-7274-90e33e" Oct 30 18:26:04.440867 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:04.440824 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74525\" (UniqueName: \"kubernetes.io/projected/ae3b1f01-05b6-469f-9966-1336b599f53e-kube-api-access-74525\") pod \"test-curl-7274-90e33e\" (UID: \"ae3b1f01-05b6-469f-9966-1336b599f53e\") " pod="opendatahub/test-curl-7274-90e33e" Oct 30 18:26:04.617559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:04.617482 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7274-90e33e" Oct 30 18:26:04.972642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:04.972555 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x" event={"ID":"598f6936-6be8-480f-b11f-62fc8bc62c07","Type":"ContainerDied","Data":"97ae47a976812e07a85a90f9e7d486e51dd7159689a4faffce6ec3c82632e0f6"} Oct 30 18:26:04.973146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:04.971944 5232 generic.go:356] "Generic (PLEG): container finished" podID="598f6936-6be8-480f-b11f-62fc8bc62c07" containerID="97ae47a976812e07a85a90f9e7d486e51dd7159689a4faffce6ec3c82632e0f6" exitCode=0 Oct 30 18:26:05.060334 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:05.046284 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7274-90e33e"] Oct 30 18:26:05.060334 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:26:05.049778 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podae3b1f01_05b6_469f_9966_1336b599f53e.slice/crio-8a8a5a83f3c2ac6d20933a8d1433fcca584e6fe20c431d3c62c7ba9f70065acd WatchSource:0}: Error finding container 8a8a5a83f3c2ac6d20933a8d1433fcca584e6fe20c431d3c62c7ba9f70065acd: Status 404 returned error can't find the container with id 8a8a5a83f3c2ac6d20933a8d1433fcca584e6fe20c431d3c62c7ba9f70065acd Oct 30 18:26:05.640085 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:05.640036 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw"] Oct 30 18:26:05.993349 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:05.993305 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7274-90e33e" event={"ID":"ae3b1f01-05b6-469f-9966-1336b599f53e","Type":"ContainerStarted","Data":"8a8a5a83f3c2ac6d20933a8d1433fcca584e6fe20c431d3c62c7ba9f70065acd"} Oct 30 18:26:06.162044 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:06.162010 5232 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a994e2_22c0_49ce_8d6b_1898bf2f7edc.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:26:06.167165 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:06.167119 5232 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a994e2_22c0_49ce_8d6b_1898bf2f7edc.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:26:06.668888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:06.668854 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x" Oct 30 18:26:06.835014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:06.834927 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/598f6936-6be8-480f-b11f-62fc8bc62c07-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/598f6936-6be8-480f-b11f-62fc8bc62c07-maas-api-openshift-ca\") pod \"598f6936-6be8-480f-b11f-62fc8bc62c07\" (UID: \"598f6936-6be8-480f-b11f-62fc8bc62c07\") " Oct 30 18:26:06.835014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:06.834991 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/598f6936-6be8-480f-b11f-62fc8bc62c07-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/598f6936-6be8-480f-b11f-62fc8bc62c07-maas-api-xks-ca\") pod \"598f6936-6be8-480f-b11f-62fc8bc62c07\" (UID: \"598f6936-6be8-480f-b11f-62fc8bc62c07\") " Oct 30 18:26:06.835341 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:06.835021 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/598f6936-6be8-480f-b11f-62fc8bc62c07-kube-api-access-n75kt\" (UniqueName: \"kubernetes.io/projected/598f6936-6be8-480f-b11f-62fc8bc62c07-kube-api-access-n75kt\") pod \"598f6936-6be8-480f-b11f-62fc8bc62c07\" (UID: \"598f6936-6be8-480f-b11f-62fc8bc62c07\") " Oct 30 18:26:06.836871 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:06.836842 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/598f6936-6be8-480f-b11f-62fc8bc62c07-maas-api-openshift-ca" pod "598f6936-6be8-480f-b11f-62fc8bc62c07" (UID: "598f6936-6be8-480f-b11f-62fc8bc62c07"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:06.840203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:06.840176 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/598f6936-6be8-480f-b11f-62fc8bc62c07-maas-api-xks-ca" pod "598f6936-6be8-480f-b11f-62fc8bc62c07" (UID: "598f6936-6be8-480f-b11f-62fc8bc62c07"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:06.840433 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:06.840411 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/598f6936-6be8-480f-b11f-62fc8bc62c07-kube-api-access-n75kt" pod "598f6936-6be8-480f-b11f-62fc8bc62c07" (UID: "598f6936-6be8-480f-b11f-62fc8bc62c07"). InnerVolumeSpecName "kube-api-access-n75kt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:06.939723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:06.938114 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/598f6936-6be8-480f-b11f-62fc8bc62c07-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:06.939723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:06.938158 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/598f6936-6be8-480f-b11f-62fc8bc62c07-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:06.939723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:06.938175 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n75kt\" (UniqueName: \"kubernetes.io/projected/598f6936-6be8-480f-b11f-62fc8bc62c07-kube-api-access-n75kt\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:07.006850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:07.006811 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x" event={"ID":"598f6936-6be8-480f-b11f-62fc8bc62c07","Type":"ContainerDied","Data":"0fcb8416c4c91aed2371775f1e8d0f3a3ff2340ab2c770d3b27cf8d894054072"} Oct 30 18:26:07.006850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:07.006839 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x" Oct 30 18:26:07.007457 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:07.006878 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fcb8416c4c91aed2371775f1e8d0f3a3ff2340ab2c770d3b27cf8d894054072" Oct 30 18:26:09.356862 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:09.210627 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:26:09.356862 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:09.210686 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.175:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:09.356862 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:09.239639 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:26:09.356862 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:09.239759 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" podUID="352fefd4-5ce5-4ce2-81ff-28769154e70b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.178:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:10.035587 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.035539 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7274-90e33e" event={"ID":"ae3b1f01-05b6-469f-9966-1336b599f53e","Type":"ContainerStarted","Data":"66fe81f40bbaf175e84e19c6619f5610446d27852d8185b90d01223c8b5f12e3"} Oct 30 18:26:10.075124 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.075074 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-7274-90e33e" podStartSLOduration=1.738582642 podStartE2EDuration="6.075060746s" totalImagesPullingTime="4.336478104s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:10.073220193 +0000 UTC m=+1469.389640515" watchObservedRunningTime="2026-10-30 18:26:10.075060746 +0000 UTC m=+1469.391481067" Oct 30 18:26:10.432150 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.335968 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.179:15021/healthz/ready\": read tcp 10.133.0.2:42454->10.133.0.179:15021: read: connection reset by peer" start-of-body= Oct 30 18:26:10.432150 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.336027 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" podUID="bf05a858-37f7-4ce6-a4ba-5431a372d1b3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.179:15021/healthz/ready\": read tcp 10.133.0.2:42454->10.133.0.179:15021: read: connection reset by peer" Oct 30 18:26:10.470840 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:10.470799 5232 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a994e2_22c0_49ce_8d6b_1898bf2f7edc.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:26:10.531932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.529320 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:26:10.531932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.529381 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" podUID="d1fd255d-3e8c-489a-9151-cb9533af2405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.177:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:10.631094 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.631054 5232 patch_prober.go:29] interesting pod/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:26:10.631326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.631136 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:10.641108 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.641060 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw" podUID="06ee5406-0fa7-4e83-a199-9fc329adba3b" containerName="payload-processing" containerID="cri-o://6c3ac1908aaf13cde5ba9b289e97fafa866e7d697e64daf1ece0ab562b52c62a" gracePeriod=25 Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.777704 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.806256 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-kube-api-access-n89s5\" (UniqueName: \"kubernetes.io/projected/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-kube-api-access-n89s5\") pod \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.806333 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istiod-ca-cert\") pod \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.806418 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-token\" (UniqueName: \"kubernetes.io/projected/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-token\") pod \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.806466 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-workload-socket\") pod \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.806535 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-envoy\") pod \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.806590 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-workload-certs\") pod \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.806664 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-data\") pod \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.806731 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-credential-socket\") pod \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.806779 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-podinfo\") pod \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\" (UID: \"bf05a858-37f7-4ce6-a4ba-5431a372d1b3\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.807157 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-workload-socket" pod "bf05a858-37f7-4ce6-a4ba-5431a372d1b3" (UID: "bf05a858-37f7-4ce6-a4ba-5431a372d1b3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.807450 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.808734 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istiod-ca-cert" pod "bf05a858-37f7-4ce6-a4ba-5431a372d1b3" (UID: "bf05a858-37f7-4ce6-a4ba-5431a372d1b3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.809613 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-credential-socket" pod "bf05a858-37f7-4ce6-a4ba-5431a372d1b3" (UID: "bf05a858-37f7-4ce6-a4ba-5431a372d1b3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.809661 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.809664 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-workload-certs" pod "bf05a858-37f7-4ce6-a4ba-5431a372d1b3" (UID: "bf05a858-37f7-4ce6-a4ba-5431a372d1b3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.810242 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-data" pod "bf05a858-37f7-4ce6-a4ba-5431a372d1b3" (UID: "bf05a858-37f7-4ce6-a4ba-5431a372d1b3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.814502 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-token" pod "bf05a858-37f7-4ce6-a4ba-5431a372d1b3" (UID: "bf05a858-37f7-4ce6-a4ba-5431a372d1b3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.843751 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-kube-api-access-n89s5" pod "bf05a858-37f7-4ce6-a4ba-5431a372d1b3" (UID: "bf05a858-37f7-4ce6-a4ba-5431a372d1b3"). InnerVolumeSpecName "kube-api-access-n89s5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.843882 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-envoy" pod "bf05a858-37f7-4ce6-a4ba-5431a372d1b3" (UID: "bf05a858-37f7-4ce6-a4ba-5431a372d1b3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.843982 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-podinfo" pod "bf05a858-37f7-4ce6-a4ba-5431a372d1b3" (UID: "bf05a858-37f7-4ce6-a4ba-5431a372d1b3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.907964 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-workload-certs\") pod \"352fefd4-5ce5-4ce2-81ff-28769154e70b\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.908016 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-workload-socket\") pod \"352fefd4-5ce5-4ce2-81ff-28769154e70b\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.908218 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-podinfo\") pod \"352fefd4-5ce5-4ce2-81ff-28769154e70b\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.908259 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-credential-socket\") pod \"352fefd4-5ce5-4ce2-81ff-28769154e70b\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.908321 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/352fefd4-5ce5-4ce2-81ff-28769154e70b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352fefd4-5ce5-4ce2-81ff-28769154e70b-istiod-ca-cert\") pod \"352fefd4-5ce5-4ce2-81ff-28769154e70b\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.908389 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-envoy\") pod \"352fefd4-5ce5-4ce2-81ff-28769154e70b\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.908430 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-data\") pod \"352fefd4-5ce5-4ce2-81ff-28769154e70b\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.908475 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-token\" (UniqueName: \"kubernetes.io/projected/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-token\") pod \"352fefd4-5ce5-4ce2-81ff-28769154e70b\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.908686 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/352fefd4-5ce5-4ce2-81ff-28769154e70b-kube-api-access-gczwr\" (UniqueName: \"kubernetes.io/projected/352fefd4-5ce5-4ce2-81ff-28769154e70b-kube-api-access-gczwr\") pod \"352fefd4-5ce5-4ce2-81ff-28769154e70b\" (UID: \"352fefd4-5ce5-4ce2-81ff-28769154e70b\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.908854 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-credential-socket" pod "352fefd4-5ce5-4ce2-81ff-28769154e70b" (UID: "352fefd4-5ce5-4ce2-81ff-28769154e70b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.909089 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-workload-certs" pod "352fefd4-5ce5-4ce2-81ff-28769154e70b" (UID: "352fefd4-5ce5-4ce2-81ff-28769154e70b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.909265 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-workload-socket" pod "352fefd4-5ce5-4ce2-81ff-28769154e70b" (UID: "352fefd4-5ce5-4ce2-81ff-28769154e70b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.910059 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-data" pod "352fefd4-5ce5-4ce2-81ff-28769154e70b" (UID: "352fefd4-5ce5-4ce2-81ff-28769154e70b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.910566 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.910587 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.910720 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.910737 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.910751 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.910765 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n89s5\" (UniqueName: \"kubernetes.io/projected/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-kube-api-access-n89s5\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.910780 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.910793 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.910806 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.910819 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.910832 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.910846 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bf05a858-37f7-4ce6-a4ba-5431a372d1b3-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.910998 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/352fefd4-5ce5-4ce2-81ff-28769154e70b-istiod-ca-cert" pod "352fefd4-5ce5-4ce2-81ff-28769154e70b" (UID: "352fefd4-5ce5-4ce2-81ff-28769154e70b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.929918 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-podinfo" pod "352fefd4-5ce5-4ce2-81ff-28769154e70b" (UID: "352fefd4-5ce5-4ce2-81ff-28769154e70b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.930066 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352fefd4-5ce5-4ce2-81ff-28769154e70b-kube-api-access-gczwr" pod "352fefd4-5ce5-4ce2-81ff-28769154e70b" (UID: "352fefd4-5ce5-4ce2-81ff-28769154e70b"). InnerVolumeSpecName "kube-api-access-gczwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.940600 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-token" pod "352fefd4-5ce5-4ce2-81ff-28769154e70b" (UID: "352fefd4-5ce5-4ce2-81ff-28769154e70b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:10.951161 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-envoy" pod "352fefd4-5ce5-4ce2-81ff-28769154e70b" (UID: "352fefd4-5ce5-4ce2-81ff-28769154e70b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.012318 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/352fefd4-5ce5-4ce2-81ff-28769154e70b-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.012342 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.012350 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.012363 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gczwr\" (UniqueName: \"kubernetes.io/projected/352fefd4-5ce5-4ce2-81ff-28769154e70b-kube-api-access-gczwr\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.012375 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/352fefd4-5ce5-4ce2-81ff-28769154e70b-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.030921 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.036367 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7274-90e33e"] Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.042221 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7274-90e33e"] Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.049238 5232 generic.go:356] "Generic (PLEG): container finished" podID="352fefd4-5ce5-4ce2-81ff-28769154e70b" containerID="e9ed2e2dea302a6b56108124bc5cc3debc43fe6a6f0700f69de5c79b309b010d" exitCode=0 Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.049390 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" event={"ID":"352fefd4-5ce5-4ce2-81ff-28769154e70b","Type":"ContainerDied","Data":"e9ed2e2dea302a6b56108124bc5cc3debc43fe6a6f0700f69de5c79b309b010d"} Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.049444 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" event={"ID":"352fefd4-5ce5-4ce2-81ff-28769154e70b","Type":"ContainerDied","Data":"9e2ac10b0dff16cf351ca51062109e9060c6ad5092114fbba3c0becf000876f7"} Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.049474 5232 scope.go:122] "RemoveContainer" containerID="e9ed2e2dea302a6b56108124bc5cc3debc43fe6a6f0700f69de5c79b309b010d" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.049672 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.113049 5232 generic.go:356] "Generic (PLEG): container finished" podID="06ee5406-0fa7-4e83-a199-9fc329adba3b" containerID="6c3ac1908aaf13cde5ba9b289e97fafa866e7d697e64daf1ece0ab562b52c62a" exitCode=0 Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.113208 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw" event={"ID":"06ee5406-0fa7-4e83-a199-9fc329adba3b","Type":"ContainerDied","Data":"6c3ac1908aaf13cde5ba9b289e97fafa866e7d697e64daf1ece0ab562b52c62a"} Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.113265 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw" event={"ID":"06ee5406-0fa7-4e83-a199-9fc329adba3b","Type":"ContainerDied","Data":"a4f286c6ce34f6e9b9f2e2079e28d58ae8b68f160dc2f8053fb550493c0e1d41"} Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.113359 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.114120 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ee5406-0fa7-4e83-a199-9fc329adba3b-kube-api-access-vc9v9\" (UniqueName: \"kubernetes.io/projected/06ee5406-0fa7-4e83-a199-9fc329adba3b-kube-api-access-vc9v9\") pod \"06ee5406-0fa7-4e83-a199-9fc329adba3b\" (UID: \"06ee5406-0fa7-4e83-a199-9fc329adba3b\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.114212 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ee5406-0fa7-4e83-a199-9fc329adba3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ee5406-0fa7-4e83-a199-9fc329adba3b-plugins-config-volume\") pod \"06ee5406-0fa7-4e83-a199-9fc329adba3b\" (UID: \"06ee5406-0fa7-4e83-a199-9fc329adba3b\") " Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.116187 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ee5406-0fa7-4e83-a199-9fc329adba3b-plugins-config-volume" pod "06ee5406-0fa7-4e83-a199-9fc329adba3b" (UID: "06ee5406-0fa7-4e83-a199-9fc329adba3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.122246 5232 generic.go:356] "Generic (PLEG): container finished" podID="bf05a858-37f7-4ce6-a4ba-5431a372d1b3" containerID="d80b9482c4286db5bd9b75ee0ce8bd8f826ff6b8d324580e6a2f7d12dcd678fb" exitCode=0 Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.122387 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" event={"ID":"bf05a858-37f7-4ce6-a4ba-5431a372d1b3","Type":"ContainerDied","Data":"d80b9482c4286db5bd9b75ee0ce8bd8f826ff6b8d324580e6a2f7d12dcd678fb"} Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.122446 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" event={"ID":"bf05a858-37f7-4ce6-a4ba-5431a372d1b3","Type":"ContainerDied","Data":"61f585b92473d487e8c81e44cafbfa6225b80d4110cd0d2057416040ea47eb40"} Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.122634 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.151667 5232 generic.go:356] "Generic (PLEG): container finished" podID="d1fd255d-3e8c-489a-9151-cb9533af2405" containerID="2c34e13eac894b19ed5ae92d3d22c57afd4637ac1255e4a25e2dccc5862ad943" exitCode=0 Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.151837 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" event={"ID":"d1fd255d-3e8c-489a-9151-cb9533af2405","Type":"ContainerDied","Data":"2c34e13eac894b19ed5ae92d3d22c57afd4637ac1255e4a25e2dccc5862ad943"} Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.192700 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ee5406-0fa7-4e83-a199-9fc329adba3b-kube-api-access-vc9v9" pod "06ee5406-0fa7-4e83-a199-9fc329adba3b" (UID: "06ee5406-0fa7-4e83-a199-9fc329adba3b"). InnerVolumeSpecName "kube-api-access-vc9v9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.192759 5232 scope.go:122] "RemoveContainer" containerID="e9ed2e2dea302a6b56108124bc5cc3debc43fe6a6f0700f69de5c79b309b010d" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:11.194900 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e9ed2e2dea302a6b56108124bc5cc3debc43fe6a6f0700f69de5c79b309b010d\": container with ID starting with e9ed2e2dea302a6b56108124bc5cc3debc43fe6a6f0700f69de5c79b309b010d not found: ID does not exist" containerID="e9ed2e2dea302a6b56108124bc5cc3debc43fe6a6f0700f69de5c79b309b010d" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.194949 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e9ed2e2dea302a6b56108124bc5cc3debc43fe6a6f0700f69de5c79b309b010d"} err="failed to get container status \"e9ed2e2dea302a6b56108124bc5cc3debc43fe6a6f0700f69de5c79b309b010d\": rpc error: code = NotFound desc = could not find container \"e9ed2e2dea302a6b56108124bc5cc3debc43fe6a6f0700f69de5c79b309b010d\": container with ID starting with e9ed2e2dea302a6b56108124bc5cc3debc43fe6a6f0700f69de5c79b309b010d not found: ID does not exist" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.194984 5232 scope.go:122] "RemoveContainer" containerID="6c3ac1908aaf13cde5ba9b289e97fafa866e7d697e64daf1ece0ab562b52c62a" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.196170 5232 generic.go:356] "Generic (PLEG): container finished" podID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerID="7ef1400449bd017435f6b7ce923a64cb64a6145efafc6015ec46f4e75e959766" exitCode=0 Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.196303 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" event={"ID":"7623b1d3-8da3-4971-b1ba-486f5e1f8db2","Type":"ContainerDied","Data":"7ef1400449bd017435f6b7ce923a64cb64a6145efafc6015ec46f4e75e959766"} Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.216572 5232 scope.go:122] "RemoveContainer" containerID="6c3ac1908aaf13cde5ba9b289e97fafa866e7d697e64daf1ece0ab562b52c62a" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:11.219085 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c3ac1908aaf13cde5ba9b289e97fafa866e7d697e64daf1ece0ab562b52c62a\": container with ID starting with 6c3ac1908aaf13cde5ba9b289e97fafa866e7d697e64daf1ece0ab562b52c62a not found: ID does not exist" containerID="6c3ac1908aaf13cde5ba9b289e97fafa866e7d697e64daf1ece0ab562b52c62a" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.219126 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c3ac1908aaf13cde5ba9b289e97fafa866e7d697e64daf1ece0ab562b52c62a"} err="failed to get container status \"6c3ac1908aaf13cde5ba9b289e97fafa866e7d697e64daf1ece0ab562b52c62a\": rpc error: code = NotFound desc = could not find container \"6c3ac1908aaf13cde5ba9b289e97fafa866e7d697e64daf1ece0ab562b52c62a\": container with ID starting with 6c3ac1908aaf13cde5ba9b289e97fafa866e7d697e64daf1ece0ab562b52c62a not found: ID does not exist" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.219153 5232 scope.go:122] "RemoveContainer" containerID="d80b9482c4286db5bd9b75ee0ce8bd8f826ff6b8d324580e6a2f7d12dcd678fb" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.219583 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vc9v9\" (UniqueName: \"kubernetes.io/projected/06ee5406-0fa7-4e83-a199-9fc329adba3b-kube-api-access-vc9v9\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.219611 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06ee5406-0fa7-4e83-a199-9fc329adba3b-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.224568 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg"] Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.227776 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-f7xdg"] Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.241237 5232 scope.go:122] "RemoveContainer" containerID="d80b9482c4286db5bd9b75ee0ce8bd8f826ff6b8d324580e6a2f7d12dcd678fb" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:11.241678 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80b9482c4286db5bd9b75ee0ce8bd8f826ff6b8d324580e6a2f7d12dcd678fb\": container with ID starting with d80b9482c4286db5bd9b75ee0ce8bd8f826ff6b8d324580e6a2f7d12dcd678fb not found: ID does not exist" containerID="d80b9482c4286db5bd9b75ee0ce8bd8f826ff6b8d324580e6a2f7d12dcd678fb" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.241705 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80b9482c4286db5bd9b75ee0ce8bd8f826ff6b8d324580e6a2f7d12dcd678fb"} err="failed to get container status \"d80b9482c4286db5bd9b75ee0ce8bd8f826ff6b8d324580e6a2f7d12dcd678fb\": rpc error: code = NotFound desc = could not find container \"d80b9482c4286db5bd9b75ee0ce8bd8f826ff6b8d324580e6a2f7d12dcd678fb\": container with ID starting with d80b9482c4286db5bd9b75ee0ce8bd8f826ff6b8d324580e6a2f7d12dcd678fb not found: ID does not exist" Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.292907 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n"] Oct 30 18:26:11.314783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.299058 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-z8g6n"] Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.358665 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6"] Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.359645 5232 cpu_manager.go:414] "containerMap: removing container" podUID="598f6936-6be8-480f-b11f-62fc8bc62c07" containerName="revoke-keys" Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.359675 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="598f6936-6be8-480f-b11f-62fc8bc62c07" containerName="revoke-keys" Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.359693 5232 cpu_manager.go:414] "containerMap: removing container" podUID="352fefd4-5ce5-4ce2-81ff-28769154e70b" containerName="istio-proxy" Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.359700 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="352fefd4-5ce5-4ce2-81ff-28769154e70b" containerName="istio-proxy" Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.359726 5232 cpu_manager.go:414] "containerMap: removing container" podUID="ae3b1f01-05b6-469f-9966-1336b599f53e" containerName="test-curl-7274-90e33e" Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.359731 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae3b1f01-05b6-469f-9966-1336b599f53e" containerName="test-curl-7274-90e33e" Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.359771 5232 cpu_manager.go:414] "containerMap: removing container" podUID="06ee5406-0fa7-4e83-a199-9fc329adba3b" containerName="payload-processing" Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.359779 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ee5406-0fa7-4e83-a199-9fc329adba3b" containerName="payload-processing" Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.359812 5232 cpu_manager.go:414] "containerMap: removing container" podUID="bf05a858-37f7-4ce6-a4ba-5431a372d1b3" containerName="istio-proxy" Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.359820 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf05a858-37f7-4ce6-a4ba-5431a372d1b3" containerName="istio-proxy" Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.359990 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae3b1f01-05b6-469f-9966-1336b599f53e" containerName="test-curl-7274-90e33e" Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.360012 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ee5406-0fa7-4e83-a199-9fc329adba3b" containerName="payload-processing" Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.360023 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf05a858-37f7-4ce6-a4ba-5431a372d1b3" containerName="istio-proxy" Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.360032 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="598f6936-6be8-480f-b11f-62fc8bc62c07" containerName="revoke-keys" Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.360043 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="352fefd4-5ce5-4ce2-81ff-28769154e70b" containerName="istio-proxy" Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.370260 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.371260 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6" Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.383959 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6"] Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.388787 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:26:11.393696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.388847 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:26:11.426959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.425235 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-envoy\") pod \"d1fd255d-3e8c-489a-9151-cb9533af2405\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " Oct 30 18:26:11.426959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.425310 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d1fd255d-3e8c-489a-9151-cb9533af2405-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1fd255d-3e8c-489a-9151-cb9533af2405-istiod-ca-cert\") pod \"d1fd255d-3e8c-489a-9151-cb9533af2405\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " Oct 30 18:26:11.426959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.425541 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-workload-certs\") pod \"d1fd255d-3e8c-489a-9151-cb9533af2405\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " Oct 30 18:26:11.426959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.425619 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-credential-socket\") pod \"d1fd255d-3e8c-489a-9151-cb9533af2405\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " Oct 30 18:26:11.426959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.425764 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1fd255d-3e8c-489a-9151-cb9533af2405-kube-api-access-btkgv\" (UniqueName: \"kubernetes.io/projected/d1fd255d-3e8c-489a-9151-cb9533af2405-kube-api-access-btkgv\") pod \"d1fd255d-3e8c-489a-9151-cb9533af2405\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " Oct 30 18:26:11.426959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.425802 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-workload-socket\") pod \"d1fd255d-3e8c-489a-9151-cb9533af2405\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " Oct 30 18:26:11.426959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.425832 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-token\" (UniqueName: \"kubernetes.io/projected/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-token\") pod \"d1fd255d-3e8c-489a-9151-cb9533af2405\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " Oct 30 18:26:11.426959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.425859 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-data\") pod \"d1fd255d-3e8c-489a-9151-cb9533af2405\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " Oct 30 18:26:11.426959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.426826 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d1fd255d-3e8c-489a-9151-cb9533af2405-istiod-ca-cert" pod "d1fd255d-3e8c-489a-9151-cb9533af2405" (UID: "d1fd255d-3e8c-489a-9151-cb9533af2405"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:11.428255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.427091 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-workload-socket" pod "d1fd255d-3e8c-489a-9151-cb9533af2405" (UID: "d1fd255d-3e8c-489a-9151-cb9533af2405"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.428255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.427391 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-credential-socket" pod "d1fd255d-3e8c-489a-9151-cb9533af2405" (UID: "d1fd255d-3e8c-489a-9151-cb9533af2405"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.428255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.427448 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-workload-certs" pod "d1fd255d-3e8c-489a-9151-cb9533af2405" (UID: "d1fd255d-3e8c-489a-9151-cb9533af2405"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.428255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.427728 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-data" pod "d1fd255d-3e8c-489a-9151-cb9533af2405" (UID: "d1fd255d-3e8c-489a-9151-cb9533af2405"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.428255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.427812 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-podinfo\") pod \"d1fd255d-3e8c-489a-9151-cb9533af2405\" (UID: \"d1fd255d-3e8c-489a-9151-cb9533af2405\") " Oct 30 18:26:11.428255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.428094 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4cnmf\" (UniqueName: \"kubernetes.io/projected/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-kube-api-access-4cnmf\") pod \"maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6\" (UID: \"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6" Oct 30 18:26:11.428255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.428164 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6\" (UID: \"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6" Oct 30 18:26:11.428831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.428442 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6\" (UID: \"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6" Oct 30 18:26:11.428831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.428633 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d1fd255d-3e8c-489a-9151-cb9533af2405-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.428831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.428649 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.428831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.428661 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.428831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.428676 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.428831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.428689 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.434963 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.434936 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-envoy" pod "d1fd255d-3e8c-489a-9151-cb9533af2405" (UID: "d1fd255d-3e8c-489a-9151-cb9533af2405"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.435401 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.435379 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-podinfo" pod "d1fd255d-3e8c-489a-9151-cb9533af2405" (UID: "d1fd255d-3e8c-489a-9151-cb9533af2405"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:11.435758 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.435733 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-token" pod "d1fd255d-3e8c-489a-9151-cb9533af2405" (UID: "d1fd255d-3e8c-489a-9151-cb9533af2405"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:11.446017 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.445992 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d1fd255d-3e8c-489a-9151-cb9533af2405-kube-api-access-btkgv" pod "d1fd255d-3e8c-489a-9151-cb9533af2405" (UID: "d1fd255d-3e8c-489a-9151-cb9533af2405"). InnerVolumeSpecName "kube-api-access-btkgv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:11.510375 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.510346 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw"] Oct 30 18:26:11.514199 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.514171 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-295497-7878b9c7d6-qd2xw"] Oct 30 18:26:11.530222 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.530189 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4cnmf\" (UniqueName: \"kubernetes.io/projected/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-kube-api-access-4cnmf\") pod \"maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6\" (UID: \"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6" Oct 30 18:26:11.530331 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.530269 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6\" (UID: \"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6" Oct 30 18:26:11.530523 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.530493 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6\" (UID: \"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6" Oct 30 18:26:11.530642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.530626 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btkgv\" (UniqueName: \"kubernetes.io/projected/d1fd255d-3e8c-489a-9151-cb9533af2405-kube-api-access-btkgv\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.530682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.530649 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.530682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.530663 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.530682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.530677 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d1fd255d-3e8c-489a-9151-cb9533af2405-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.531102 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.531083 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6\" (UID: \"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6" Oct 30 18:26:11.532891 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.532872 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6\" (UID: \"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6" Oct 30 18:26:11.557571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.557494 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4cnmf\" (UniqueName: \"kubernetes.io/projected/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-kube-api-access-4cnmf\") pod \"maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6\" (UID: \"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6" Oct 30 18:26:11.606377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.606342 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:26:11.632906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.631995 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-workload-socket\") pod \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " Oct 30 18:26:11.632906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.632061 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-podinfo\") pod \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " Oct 30 18:26:11.632906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.632163 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-data\") pod \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " Oct 30 18:26:11.632906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.632224 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-envoy\") pod \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " Oct 30 18:26:11.632906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.632251 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-workload-certs\") pod \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " Oct 30 18:26:11.632906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.632275 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-credential-socket\") pod \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " Oct 30 18:26:11.632906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.632302 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-token\" (UniqueName: \"kubernetes.io/projected/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-token\") pod \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " Oct 30 18:26:11.632906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.632350 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istiod-ca-cert\") pod \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " Oct 30 18:26:11.632906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.632460 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-kube-api-access-5mvs6\" (UniqueName: \"kubernetes.io/projected/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-kube-api-access-5mvs6\") pod \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\" (UID: \"7623b1d3-8da3-4971-b1ba-486f5e1f8db2\") " Oct 30 18:26:11.632906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.632586 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-workload-socket" pod "7623b1d3-8da3-4971-b1ba-486f5e1f8db2" (UID: "7623b1d3-8da3-4971-b1ba-486f5e1f8db2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.632906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.632722 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-data" pod "7623b1d3-8da3-4971-b1ba-486f5e1f8db2" (UID: "7623b1d3-8da3-4971-b1ba-486f5e1f8db2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.632906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.632852 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-credential-socket" pod "7623b1d3-8da3-4971-b1ba-486f5e1f8db2" (UID: "7623b1d3-8da3-4971-b1ba-486f5e1f8db2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.633889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.632957 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-workload-certs" pod "7623b1d3-8da3-4971-b1ba-486f5e1f8db2" (UID: "7623b1d3-8da3-4971-b1ba-486f5e1f8db2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.633889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.633414 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.633889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.633435 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.633889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.633448 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.633889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.633461 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.634479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.634442 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istiod-ca-cert" pod "7623b1d3-8da3-4971-b1ba-486f5e1f8db2" (UID: "7623b1d3-8da3-4971-b1ba-486f5e1f8db2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:11.638922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.638894 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-envoy" pod "7623b1d3-8da3-4971-b1ba-486f5e1f8db2" (UID: "7623b1d3-8da3-4971-b1ba-486f5e1f8db2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.639691 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.639666 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-token" pod "7623b1d3-8da3-4971-b1ba-486f5e1f8db2" (UID: "7623b1d3-8da3-4971-b1ba-486f5e1f8db2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:11.640773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.640733 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-podinfo" pod "7623b1d3-8da3-4971-b1ba-486f5e1f8db2" (UID: "7623b1d3-8da3-4971-b1ba-486f5e1f8db2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:11.640922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.640903 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-kube-api-access-5mvs6" pod "7623b1d3-8da3-4971-b1ba-486f5e1f8db2" (UID: "7623b1d3-8da3-4971-b1ba-486f5e1f8db2"). InnerVolumeSpecName "kube-api-access-5mvs6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:11.697660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.697623 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6" Oct 30 18:26:11.750531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.750472 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.750706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.750534 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.750706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.750551 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.750706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.750564 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.750706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.750579 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mvs6\" (UniqueName: \"kubernetes.io/projected/7623b1d3-8da3-4971-b1ba-486f5e1f8db2-kube-api-access-5mvs6\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.816677 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.816643 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:26:11.851497 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.851417 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-podinfo\") pod \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " Oct 30 18:26:11.851667 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.851565 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-data\") pod \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " Oct 30 18:26:11.851667 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.851619 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-envoy\") pod \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " Oct 30 18:26:11.851742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.851691 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-credential-socket\") pod \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " Oct 30 18:26:11.851894 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.851830 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-workload-certs\") pod \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " Oct 30 18:26:11.851894 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.851875 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4223b02-22ba-4893-a7ea-38dd0d712a9a-kube-api-access-2rm79\" (UniqueName: \"kubernetes.io/projected/c4223b02-22ba-4893-a7ea-38dd0d712a9a-kube-api-access-2rm79\") pod \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " Oct 30 18:26:11.851978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.851924 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-workload-socket\") pod \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " Oct 30 18:26:11.852013 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.851974 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istiod-ca-cert\") pod \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " Oct 30 18:26:11.852013 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.852001 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-token\" (UniqueName: \"kubernetes.io/projected/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-token\") pod \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\" (UID: \"c4223b02-22ba-4893-a7ea-38dd0d712a9a\") " Oct 30 18:26:11.852686 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.852660 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-workload-certs" pod "c4223b02-22ba-4893-a7ea-38dd0d712a9a" (UID: "c4223b02-22ba-4893-a7ea-38dd0d712a9a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.854181 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.852143 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-data" pod "c4223b02-22ba-4893-a7ea-38dd0d712a9a" (UID: "c4223b02-22ba-4893-a7ea-38dd0d712a9a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.854181 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.853804 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-credential-socket" pod "c4223b02-22ba-4893-a7ea-38dd0d712a9a" (UID: "c4223b02-22ba-4893-a7ea-38dd0d712a9a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.854181 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.853937 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istiod-ca-cert" pod "c4223b02-22ba-4893-a7ea-38dd0d712a9a" (UID: "c4223b02-22ba-4893-a7ea-38dd0d712a9a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:11.854398 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.854241 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-podinfo" pod "c4223b02-22ba-4893-a7ea-38dd0d712a9a" (UID: "c4223b02-22ba-4893-a7ea-38dd0d712a9a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:11.855888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.854544 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-workload-socket" pod "c4223b02-22ba-4893-a7ea-38dd0d712a9a" (UID: "c4223b02-22ba-4893-a7ea-38dd0d712a9a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.855888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.855284 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-token" pod "c4223b02-22ba-4893-a7ea-38dd0d712a9a" (UID: "c4223b02-22ba-4893-a7ea-38dd0d712a9a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:11.856139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.856038 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-envoy" pod "c4223b02-22ba-4893-a7ea-38dd0d712a9a" (UID: "c4223b02-22ba-4893-a7ea-38dd0d712a9a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:11.874844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.872397 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4223b02-22ba-4893-a7ea-38dd0d712a9a-kube-api-access-2rm79" pod "c4223b02-22ba-4893-a7ea-38dd0d712a9a" (UID: "c4223b02-22ba-4893-a7ea-38dd0d712a9a"). InnerVolumeSpecName "kube-api-access-2rm79". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:11.954300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.954261 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.954300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.954301 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.954570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.954316 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.954570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.954331 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.954570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.954374 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rm79\" (UniqueName: \"kubernetes.io/projected/c4223b02-22ba-4893-a7ea-38dd0d712a9a-kube-api-access-2rm79\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.954570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.954389 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4223b02-22ba-4893-a7ea-38dd0d712a9a-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.954570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.954403 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.954570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.954415 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:11.954570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:11.954427 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4223b02-22ba-4893-a7ea-38dd0d712a9a-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:12.168042 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.168014 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6"] Oct 30 18:26:12.208981 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.208948 5232 generic.go:356] "Generic (PLEG): container finished" podID="43d92d4d-f62f-4b68-a9bf-89186482a244" containerID="13d441f3a5c46ec1e0f3f3e38fc0f8692d40a5ce5bcd6d95ba64eeb47729137d" exitCode=0 Oct 30 18:26:12.209129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.209021 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" event={"ID":"43d92d4d-f62f-4b68-a9bf-89186482a244","Type":"ContainerDied","Data":"13d441f3a5c46ec1e0f3f3e38fc0f8692d40a5ce5bcd6d95ba64eeb47729137d"} Oct 30 18:26:12.209594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.209575 5232 scope.go:122] "RemoveContainer" containerID="13d441f3a5c46ec1e0f3f3e38fc0f8692d40a5ce5bcd6d95ba64eeb47729137d" Oct 30 18:26:12.211606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.211560 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:26:12.211721 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.211661 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:26:12.211774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.211731 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:26:12.212111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.212090 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" Oct 30 18:26:12.212111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.212105 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s" event={"ID":"d1fd255d-3e8c-489a-9151-cb9533af2405","Type":"ContainerDied","Data":"4b1a97298b0a8027e4fcfe091980ed255740466aa32814ca7e7a0b043416c739"} Oct 30 18:26:12.212296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.212154 5232 scope.go:122] "RemoveContainer" containerID="2c34e13eac894b19ed5ae92d3d22c57afd4637ac1255e4a25e2dccc5862ad943" Oct 30 18:26:12.214107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.214072 5232 generic.go:356] "Generic (PLEG): container finished" podID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerID="e0aa7425dbd1d5ab06058423414ddc13e5760d0762cf4367ec7a4288c5666d94" exitCode=0 Oct 30 18:26:12.214235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.214170 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" event={"ID":"c4223b02-22ba-4893-a7ea-38dd0d712a9a","Type":"ContainerDied","Data":"e0aa7425dbd1d5ab06058423414ddc13e5760d0762cf4367ec7a4288c5666d94"} Oct 30 18:26:12.214235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.214209 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" event={"ID":"c4223b02-22ba-4893-a7ea-38dd0d712a9a","Type":"ContainerDied","Data":"678c5f8bfc196e113288a23fa792d5b46cc2bbde86066d50ac6b4906704bb8a1"} Oct 30 18:26:12.214363 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.214320 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n" Oct 30 18:26:12.216922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.216903 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" Oct 30 18:26:12.217021 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.216916 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9" event={"ID":"7623b1d3-8da3-4971-b1ba-486f5e1f8db2","Type":"ContainerDied","Data":"896dbfa3125725bbf63b5cffe55ea45a71b3bf9698632edbbb07f983bad0a4b7"} Oct 30 18:26:12.218495 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.218468 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-7274-90e33e" podUID="ae3b1f01-05b6-469f-9966-1336b599f53e" containerName="test-curl-7274-90e33e" containerID="cri-o://66fe81f40bbaf175e84e19c6619f5610446d27852d8185b90d01223c8b5f12e3" gracePeriod=2 Oct 30 18:26:12.218642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.218621 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6" event={"ID":"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0","Type":"ContainerStarted","Data":"8d070e3fa96b18aa0304756778e6e773f1c2220f04fbba94c6d1462c92c823c0"} Oct 30 18:26:12.227934 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.227912 5232 scope.go:122] "RemoveContainer" containerID="e0aa7425dbd1d5ab06058423414ddc13e5760d0762cf4367ec7a4288c5666d94" Oct 30 18:26:12.238017 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.237988 5232 scope.go:122] "RemoveContainer" containerID="e0aa7425dbd1d5ab06058423414ddc13e5760d0762cf4367ec7a4288c5666d94" Oct 30 18:26:12.238301 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:12.238280 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e0aa7425dbd1d5ab06058423414ddc13e5760d0762cf4367ec7a4288c5666d94\": container with ID starting with e0aa7425dbd1d5ab06058423414ddc13e5760d0762cf4367ec7a4288c5666d94 not found: ID does not exist" containerID="e0aa7425dbd1d5ab06058423414ddc13e5760d0762cf4367ec7a4288c5666d94" Oct 30 18:26:12.238379 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.238308 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e0aa7425dbd1d5ab06058423414ddc13e5760d0762cf4367ec7a4288c5666d94"} err="failed to get container status \"e0aa7425dbd1d5ab06058423414ddc13e5760d0762cf4367ec7a4288c5666d94\": rpc error: code = NotFound desc = could not find container \"e0aa7425dbd1d5ab06058423414ddc13e5760d0762cf4367ec7a4288c5666d94\": container with ID starting with e0aa7425dbd1d5ab06058423414ddc13e5760d0762cf4367ec7a4288c5666d94 not found: ID does not exist" Oct 30 18:26:12.238379 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.238323 5232 scope.go:122] "RemoveContainer" containerID="7ef1400449bd017435f6b7ce923a64cb64a6145efafc6015ec46f4e75e959766" Oct 30 18:26:12.269687 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:12.269648 5232 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-7274-90e33e\" is forbidden: User \"system:node:ip-10-0-141-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-141-106.ec2.internal' and this object" podUID="ae3b1f01-05b6-469f-9966-1336b599f53e" pod="opendatahub/test-curl-7274-90e33e" Oct 30 18:26:12.308112 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.308048 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9"] Oct 30 18:26:12.320364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.320342 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-7m4r9"] Oct 30 18:26:12.353410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.353381 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s"] Oct 30 18:26:12.361802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.361781 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-r574s"] Oct 30 18:26:12.397496 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.397460 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n"] Oct 30 18:26:12.403550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.403491 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-9af5623f-openshift-default-66d4dbc5bf-6vm8n"] Oct 30 18:26:12.689998 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:12.689968 5232 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a994e2_22c0_49ce_8d6b_1898bf2f7edc.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:26:12.805719 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.805686 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ee5406-0fa7-4e83-a199-9fc329adba3b" path="/var/lib/kubelet/pods/06ee5406-0fa7-4e83-a199-9fc329adba3b/volumes" Oct 30 18:26:12.806374 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.806358 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="352fefd4-5ce5-4ce2-81ff-28769154e70b" path="/var/lib/kubelet/pods/352fefd4-5ce5-4ce2-81ff-28769154e70b/volumes" Oct 30 18:26:12.807041 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.807025 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" path="/var/lib/kubelet/pods/7623b1d3-8da3-4971-b1ba-486f5e1f8db2/volumes" Oct 30 18:26:12.808086 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.808069 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf05a858-37f7-4ce6-a4ba-5431a372d1b3" path="/var/lib/kubelet/pods/bf05a858-37f7-4ce6-a4ba-5431a372d1b3/volumes" Oct 30 18:26:12.808681 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.808666 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" path="/var/lib/kubelet/pods/c4223b02-22ba-4893-a7ea-38dd0d712a9a/volumes" Oct 30 18:26:12.809262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:12.809248 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d1fd255d-3e8c-489a-9151-cb9533af2405" path="/var/lib/kubelet/pods/d1fd255d-3e8c-489a-9151-cb9533af2405/volumes" Oct 30 18:26:13.228487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:13.228452 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" event={"ID":"43d92d4d-f62f-4b68-a9bf-89186482a244","Type":"ContainerStarted","Data":"95c4a97f1c15605ffa91358c2d90085d5d961e13c5251c928309c0b4fda6ddbd"} Oct 30 18:26:13.234678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:13.234651 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6" event={"ID":"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0","Type":"ContainerStarted","Data":"c435642bd2fc955ee767af96807a3a72fdcbfd9d8392f8b63bf95b3eea7d4ac8"} Oct 30 18:26:13.314103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:13.314051 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6" podStartSLOduration=2.314037059 podStartE2EDuration="2.314037059s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:13.312015824 +0000 UTC m=+1472.628436157" watchObservedRunningTime="2026-10-30 18:26:13.314037059 +0000 UTC m=+1472.630457380" Oct 30 18:26:13.836115 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:13.836074 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:26:13.837074 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:13.837047 5232 patch_prober.go:29] interesting pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:26:13.837184 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:13.837108 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" podUID="43d92d4d-f62f-4b68-a9bf-89186482a244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 18:26:14.245955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:14.245925 5232 generic.go:356] "Generic (PLEG): container finished" podID="bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0" containerID="c435642bd2fc955ee767af96807a3a72fdcbfd9d8392f8b63bf95b3eea7d4ac8" exitCode=0 Oct 30 18:26:14.246110 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:14.246015 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6" event={"ID":"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0","Type":"ContainerDied","Data":"c435642bd2fc955ee767af96807a3a72fdcbfd9d8392f8b63bf95b3eea7d4ac8"} Oct 30 18:26:14.491275 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:14.491252 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7274-90e33e" Oct 30 18:26:14.605655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:14.605624 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae3b1f01-05b6-469f-9966-1336b599f53e-kube-api-access-74525\" (UniqueName: \"kubernetes.io/projected/ae3b1f01-05b6-469f-9966-1336b599f53e-kube-api-access-74525\") pod \"ae3b1f01-05b6-469f-9966-1336b599f53e\" (UID: \"ae3b1f01-05b6-469f-9966-1336b599f53e\") " Oct 30 18:26:14.607943 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:14.607913 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae3b1f01-05b6-469f-9966-1336b599f53e-kube-api-access-74525" pod "ae3b1f01-05b6-469f-9966-1336b599f53e" (UID: "ae3b1f01-05b6-469f-9966-1336b599f53e"). InnerVolumeSpecName "kube-api-access-74525". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:14.707888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:14.707857 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74525\" (UniqueName: \"kubernetes.io/projected/ae3b1f01-05b6-469f-9966-1336b599f53e-kube-api-access-74525\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:14.891587 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:14.887125 5232 patch_prober.go:29] interesting pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:26:14.891587 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:14.887197 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" podUID="43d92d4d-f62f-4b68-a9bf-89186482a244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 18:26:14.956000 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:14.955953 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae3b1f01-05b6-469f-9966-1336b599f53e" path="/var/lib/kubelet/pods/ae3b1f01-05b6-469f-9966-1336b599f53e/volumes" Oct 30 18:26:15.254978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:15.254943 5232 generic.go:356] "Generic (PLEG): container finished" podID="ae3b1f01-05b6-469f-9966-1336b599f53e" containerID="66fe81f40bbaf175e84e19c6619f5610446d27852d8185b90d01223c8b5f12e3" exitCode=137 Oct 30 18:26:15.254978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:15.254990 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7274-90e33e" Oct 30 18:26:15.255259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:15.255048 5232 scope.go:122] "RemoveContainer" containerID="66fe81f40bbaf175e84e19c6619f5610446d27852d8185b90d01223c8b5f12e3" Oct 30 18:26:15.265794 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:15.265774 5232 scope.go:122] "RemoveContainer" containerID="66fe81f40bbaf175e84e19c6619f5610446d27852d8185b90d01223c8b5f12e3" Oct 30 18:26:15.266075 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:15.266050 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"66fe81f40bbaf175e84e19c6619f5610446d27852d8185b90d01223c8b5f12e3\": container with ID starting with 66fe81f40bbaf175e84e19c6619f5610446d27852d8185b90d01223c8b5f12e3 not found: ID does not exist" containerID="66fe81f40bbaf175e84e19c6619f5610446d27852d8185b90d01223c8b5f12e3" Oct 30 18:26:15.266142 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:15.266085 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"66fe81f40bbaf175e84e19c6619f5610446d27852d8185b90d01223c8b5f12e3"} err="failed to get container status \"66fe81f40bbaf175e84e19c6619f5610446d27852d8185b90d01223c8b5f12e3\": rpc error: code = NotFound desc = could not find container \"66fe81f40bbaf175e84e19c6619f5610446d27852d8185b90d01223c8b5f12e3\": container with ID starting with 66fe81f40bbaf175e84e19c6619f5610446d27852d8185b90d01223c8b5f12e3 not found: ID does not exist" Oct 30 18:26:15.487029 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:15.487004 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6" Oct 30 18:26:15.609438 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:15.609359 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-maas-api-xks-ca\") pod \"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0\" (UID: \"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0\") " Oct 30 18:26:15.609615 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:15.609493 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-maas-api-openshift-ca\") pod \"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0\" (UID: \"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0\") " Oct 30 18:26:15.609705 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:15.609625 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-kube-api-access-4cnmf\" (UniqueName: \"kubernetes.io/projected/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-kube-api-access-4cnmf\") pod \"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0\" (UID: \"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0\") " Oct 30 18:26:15.632534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:15.629821 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-maas-api-openshift-ca" pod "bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0" (UID: "bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:15.637733 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:15.637439 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-maas-api-xks-ca" pod "bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0" (UID: "bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:15.637962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:15.637932 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-kube-api-access-4cnmf" pod "bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0" (UID: "bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0"). InnerVolumeSpecName "kube-api-access-4cnmf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:15.734698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:15.732611 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:15.734698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:15.732650 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:15.734698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:15.732668 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4cnmf\" (UniqueName: \"kubernetes.io/projected/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0-kube-api-access-4cnmf\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:15.756969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:15.756937 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:26:15.834967 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:15.834917 5232 patch_prober.go:29] interesting pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:26:15.835170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:15.835003 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" podUID="43d92d4d-f62f-4b68-a9bf-89186482a244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 18:26:16.264591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:16.264561 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6" Oct 30 18:26:16.265029 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:16.264550 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6" event={"ID":"bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0","Type":"ContainerDied","Data":"8d070e3fa96b18aa0304756778e6e773f1c2220f04fbba94c6d1462c92c823c0"} Oct 30 18:26:16.265029 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:16.264688 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8d070e3fa96b18aa0304756778e6e773f1c2220f04fbba94c6d1462c92c823c0" Oct 30 18:26:16.832534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:16.823174 5232 patch_prober.go:29] interesting pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:26:16.832534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:16.823253 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" podUID="43d92d4d-f62f-4b68-a9bf-89186482a244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 18:26:17.834538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:17.831912 5232 patch_prober.go:29] interesting pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:26:17.834538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:17.831971 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" podUID="43d92d4d-f62f-4b68-a9bf-89186482a244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 18:26:18.832268 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:18.827951 5232 patch_prober.go:29] interesting pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" start-of-body= Oct 30 18:26:18.832268 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:18.828042 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" podUID="43d92d4d-f62f-4b68-a9bf-89186482a244" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.176:15021/healthz/ready\": dial tcp 10.133.0.176:15021: connect: connection refused" Oct 30 18:26:19.003195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:19.003162 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz"] Oct 30 18:26:19.057377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:19.055654 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f8709f-wffcz"] Oct 30 18:26:19.828597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:19.828565 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:26:19.829690 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:19.829667 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:26:20.805750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:20.805715 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="86cc0758-fb7a-404f-9016-fd00ed2b6f4b" path="/var/lib/kubelet/pods/86cc0758-fb7a-404f-9016-fd00ed2b6f4b/volumes" Oct 30 18:26:23.445935 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:23.445884 5232 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a994e2_22c0_49ce_8d6b_1898bf2f7edc.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:26:25.441258 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:25.441218 5232 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a994e2_22c0_49ce_8d6b_1898bf2f7edc.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:26:26.969761 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:26.969721 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x"] Oct 30 18:26:26.970210 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:26.970012 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" podUID="961c8dee-4505-4b84-b984-3006b6a95dd8" containerName="istio-proxy" containerID="cri-o://53ff8a18a2e8a3e875aa6adcd6d5c9c11df3a5ea338f3151839b5b7e1ba28ebd" gracePeriod=30 Oct 30 18:26:27.037326 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:27.037289 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:26:27.654271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.654239 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886"] Oct 30 18:26:27.654999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.654981 5232 cpu_manager.go:414] "containerMap: removing container" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" Oct 30 18:26:27.654999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.654999 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" Oct 30 18:26:27.655140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.655015 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" Oct 30 18:26:27.655140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.655020 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" Oct 30 18:26:27.655140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.655045 5232 cpu_manager.go:414] "containerMap: removing container" podUID="d1fd255d-3e8c-489a-9151-cb9533af2405" containerName="istio-proxy" Oct 30 18:26:27.655140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.655051 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d1fd255d-3e8c-489a-9151-cb9533af2405" containerName="istio-proxy" Oct 30 18:26:27.655140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.655063 5232 cpu_manager.go:414] "containerMap: removing container" podUID="bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0" containerName="revoke-keys" Oct 30 18:26:27.655140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.655068 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0" containerName="revoke-keys" Oct 30 18:26:27.655326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.655157 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="c4223b02-22ba-4893-a7ea-38dd0d712a9a" containerName="istio-proxy" Oct 30 18:26:27.655326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.655166 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7623b1d3-8da3-4971-b1ba-486f5e1f8db2" containerName="istio-proxy" Oct 30 18:26:27.655326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.655177 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="d1fd255d-3e8c-489a-9151-cb9533af2405" containerName="istio-proxy" Oct 30 18:26:27.655326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.655185 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0" containerName="revoke-keys" Oct 30 18:26:27.658437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.658418 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886" Oct 30 18:26:27.677937 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.677910 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886"] Oct 30 18:26:27.742744 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.742712 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:26:27.742972 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.742809 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:26:27.828124 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.827903 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d9b034-63e6-4493-94a5-43b41548d4ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18cd5a-6h886\" (UID: \"c0d9b034-63e6-4493-94a5-43b41548d4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886" Oct 30 18:26:27.828781 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.828743 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d9b034-63e6-4493-94a5-43b41548d4ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18cd5a-6h886\" (UID: \"c0d9b034-63e6-4493-94a5-43b41548d4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886" Oct 30 18:26:27.830566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.830413 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-67784\" (UniqueName: \"kubernetes.io/projected/c0d9b034-63e6-4493-94a5-43b41548d4ec-kube-api-access-67784\") pod \"maas-api-revoke-keys-e2e-api-b-18cd5a-6h886\" (UID: \"c0d9b034-63e6-4493-94a5-43b41548d4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886" Oct 30 18:26:27.932626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.932500 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d9b034-63e6-4493-94a5-43b41548d4ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18cd5a-6h886\" (UID: \"c0d9b034-63e6-4493-94a5-43b41548d4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886" Oct 30 18:26:27.932825 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.932671 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d9b034-63e6-4493-94a5-43b41548d4ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18cd5a-6h886\" (UID: \"c0d9b034-63e6-4493-94a5-43b41548d4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886" Oct 30 18:26:27.932825 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.932713 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-67784\" (UniqueName: \"kubernetes.io/projected/c0d9b034-63e6-4493-94a5-43b41548d4ec-kube-api-access-67784\") pod \"maas-api-revoke-keys-e2e-api-b-18cd5a-6h886\" (UID: \"c0d9b034-63e6-4493-94a5-43b41548d4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886" Oct 30 18:26:27.933304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.933283 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d9b034-63e6-4493-94a5-43b41548d4ec-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18cd5a-6h886\" (UID: \"c0d9b034-63e6-4493-94a5-43b41548d4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886" Oct 30 18:26:27.935127 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.935110 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d9b034-63e6-4493-94a5-43b41548d4ec-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-18cd5a-6h886\" (UID: \"c0d9b034-63e6-4493-94a5-43b41548d4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886" Oct 30 18:26:27.954985 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.954960 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-67784\" (UniqueName: \"kubernetes.io/projected/c0d9b034-63e6-4493-94a5-43b41548d4ec-kube-api-access-67784\") pod \"maas-api-revoke-keys-e2e-api-b-18cd5a-6h886\" (UID: \"c0d9b034-63e6-4493-94a5-43b41548d4ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886" Oct 30 18:26:27.971770 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:27.971744 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886" Oct 30 18:26:28.270095 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:26:28.270061 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0d9b034_63e6_4493_94a5_43b41548d4ec.slice/crio-1945e31261e33444e31897a85c4ef438252e2261f00062cfede177dd4a448342 WatchSource:0}: Error finding container 1945e31261e33444e31897a85c4ef438252e2261f00062cfede177dd4a448342: Status 404 returned error can't find the container with id 1945e31261e33444e31897a85c4ef438252e2261f00062cfede177dd4a448342 Oct 30 18:26:28.272409 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:28.272353 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886"] Oct 30 18:26:28.531024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:28.530994 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886" event={"ID":"c0d9b034-63e6-4493-94a5-43b41548d4ec","Type":"ContainerStarted","Data":"1945e31261e33444e31897a85c4ef438252e2261f00062cfede177dd4a448342"} Oct 30 18:26:29.084042 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:29.083999 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9"] Oct 30 18:26:29.088072 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:29.088046 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9" Oct 30 18:26:29.106941 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:29.106909 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9"] Oct 30 18:26:29.152925 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:29.152888 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntrtg\" (UniqueName: \"kubernetes.io/projected/36cd7371-04b2-45ad-ab88-88b3649868bb-kube-api-access-ntrtg\") pod \"maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9\" (UID: \"36cd7371-04b2-45ad-ab88-88b3649868bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9" Oct 30 18:26:29.153130 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:29.152965 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36cd7371-04b2-45ad-ab88-88b3649868bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9\" (UID: \"36cd7371-04b2-45ad-ab88-88b3649868bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9" Oct 30 18:26:29.153130 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:29.153094 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36cd7371-04b2-45ad-ab88-88b3649868bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9\" (UID: \"36cd7371-04b2-45ad-ab88-88b3649868bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9" Oct 30 18:26:29.254348 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:29.254318 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntrtg\" (UniqueName: \"kubernetes.io/projected/36cd7371-04b2-45ad-ab88-88b3649868bb-kube-api-access-ntrtg\") pod \"maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9\" (UID: \"36cd7371-04b2-45ad-ab88-88b3649868bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9" Oct 30 18:26:29.254488 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:29.254376 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36cd7371-04b2-45ad-ab88-88b3649868bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9\" (UID: \"36cd7371-04b2-45ad-ab88-88b3649868bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9" Oct 30 18:26:29.254488 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:29.254484 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36cd7371-04b2-45ad-ab88-88b3649868bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9\" (UID: \"36cd7371-04b2-45ad-ab88-88b3649868bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9" Oct 30 18:26:29.255055 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:29.255034 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36cd7371-04b2-45ad-ab88-88b3649868bb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9\" (UID: \"36cd7371-04b2-45ad-ab88-88b3649868bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9" Oct 30 18:26:29.257098 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:29.257079 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36cd7371-04b2-45ad-ab88-88b3649868bb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9\" (UID: \"36cd7371-04b2-45ad-ab88-88b3649868bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9" Oct 30 18:26:29.273749 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:29.273715 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntrtg\" (UniqueName: \"kubernetes.io/projected/36cd7371-04b2-45ad-ab88-88b3649868bb-kube-api-access-ntrtg\") pod \"maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9\" (UID: \"36cd7371-04b2-45ad-ab88-88b3649868bb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9" Oct 30 18:26:29.403591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:29.403502 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9" Oct 30 18:26:29.558572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:29.557188 5232 generic.go:356] "Generic (PLEG): container finished" podID="c0d9b034-63e6-4493-94a5-43b41548d4ec" containerID="b3108f4369363edfbfc63e3d374aa715e2964f3136fbc186d7332aa165952405" exitCode=0 Oct 30 18:26:29.558572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:29.557440 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886" event={"ID":"c0d9b034-63e6-4493-94a5-43b41548d4ec","Type":"ContainerDied","Data":"b3108f4369363edfbfc63e3d374aa715e2964f3136fbc186d7332aa165952405"} Oct 30 18:26:29.778591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:29.771892 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9"] Oct 30 18:26:30.566787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:30.566752 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9" event={"ID":"36cd7371-04b2-45ad-ab88-88b3649868bb","Type":"ContainerStarted","Data":"023b46fdc47acc056bea6340f41909a24c36219dc62239fb6bb5a145b5f1f74a"} Oct 30 18:26:30.567204 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:30.566813 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9" event={"ID":"36cd7371-04b2-45ad-ab88-88b3649868bb","Type":"ContainerStarted","Data":"d3b6cd0bcfafda2e6ae429653f4157f77adbb230f07239ceea9cc2a16534c3a3"} Oct 30 18:26:30.607618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:30.607559 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9" podStartSLOduration=1.6075445080000001 podStartE2EDuration="1.607544508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:30.604943997 +0000 UTC m=+1489.921364319" watchObservedRunningTime="2026-10-30 18:26:30.607544508 +0000 UTC m=+1489.923964829" Oct 30 18:26:30.733261 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:30.733237 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886" Oct 30 18:26:30.784892 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:30.784864 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0d9b034-63e6-4493-94a5-43b41548d4ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d9b034-63e6-4493-94a5-43b41548d4ec-maas-api-xks-ca\") pod \"c0d9b034-63e6-4493-94a5-43b41548d4ec\" (UID: \"c0d9b034-63e6-4493-94a5-43b41548d4ec\") " Oct 30 18:26:30.785089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:30.785071 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0d9b034-63e6-4493-94a5-43b41548d4ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d9b034-63e6-4493-94a5-43b41548d4ec-maas-api-openshift-ca\") pod \"c0d9b034-63e6-4493-94a5-43b41548d4ec\" (UID: \"c0d9b034-63e6-4493-94a5-43b41548d4ec\") " Oct 30 18:26:30.785171 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:30.785134 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0d9b034-63e6-4493-94a5-43b41548d4ec-kube-api-access-67784\" (UniqueName: \"kubernetes.io/projected/c0d9b034-63e6-4493-94a5-43b41548d4ec-kube-api-access-67784\") pod \"c0d9b034-63e6-4493-94a5-43b41548d4ec\" (UID: \"c0d9b034-63e6-4493-94a5-43b41548d4ec\") " Oct 30 18:26:30.785473 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:30.785447 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0d9b034-63e6-4493-94a5-43b41548d4ec-maas-api-openshift-ca" pod "c0d9b034-63e6-4493-94a5-43b41548d4ec" (UID: "c0d9b034-63e6-4493-94a5-43b41548d4ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:30.786064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:30.786035 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0d9b034-63e6-4493-94a5-43b41548d4ec-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:30.787352 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:30.787325 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0d9b034-63e6-4493-94a5-43b41548d4ec-maas-api-xks-ca" pod "c0d9b034-63e6-4493-94a5-43b41548d4ec" (UID: "c0d9b034-63e6-4493-94a5-43b41548d4ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:30.787475 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:30.787450 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0d9b034-63e6-4493-94a5-43b41548d4ec-kube-api-access-67784" pod "c0d9b034-63e6-4493-94a5-43b41548d4ec" (UID: "c0d9b034-63e6-4493-94a5-43b41548d4ec"). InnerVolumeSpecName "kube-api-access-67784". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:30.888547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:30.888499 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-67784\" (UniqueName: \"kubernetes.io/projected/c0d9b034-63e6-4493-94a5-43b41548d4ec-kube-api-access-67784\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:30.888547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:30.888550 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0d9b034-63e6-4493-94a5-43b41548d4ec-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:31.589920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:31.589889 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886" Oct 30 18:26:31.590557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:31.589907 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886" event={"ID":"c0d9b034-63e6-4493-94a5-43b41548d4ec","Type":"ContainerDied","Data":"1945e31261e33444e31897a85c4ef438252e2261f00062cfede177dd4a448342"} Oct 30 18:26:31.590877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:31.590851 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1945e31261e33444e31897a85c4ef438252e2261f00062cfede177dd4a448342" Oct 30 18:26:31.592220 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:31.592190 5232 generic.go:356] "Generic (PLEG): container finished" podID="36cd7371-04b2-45ad-ab88-88b3649868bb" containerID="023b46fdc47acc056bea6340f41909a24c36219dc62239fb6bb5a145b5f1f74a" exitCode=0 Oct 30 18:26:31.592330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:31.592241 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9" event={"ID":"36cd7371-04b2-45ad-ab88-88b3649868bb","Type":"ContainerDied","Data":"023b46fdc47acc056bea6340f41909a24c36219dc62239fb6bb5a145b5f1f74a"} Oct 30 18:26:32.256704 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.256651 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:26:32.315932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.315895 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-workload-certs\") pod \"961c8dee-4505-4b84-b984-3006b6a95dd8\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " Oct 30 18:26:32.316097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.315961 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/961c8dee-4505-4b84-b984-3006b6a95dd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961c8dee-4505-4b84-b984-3006b6a95dd8-istiod-ca-cert\") pod \"961c8dee-4505-4b84-b984-3006b6a95dd8\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " Oct 30 18:26:32.316097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.315994 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-envoy\") pod \"961c8dee-4505-4b84-b984-3006b6a95dd8\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " Oct 30 18:26:32.316097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.316074 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-data\") pod \"961c8dee-4505-4b84-b984-3006b6a95dd8\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " Oct 30 18:26:32.316261 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.316174 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-workload-certs" pod "961c8dee-4505-4b84-b984-3006b6a95dd8" (UID: "961c8dee-4505-4b84-b984-3006b6a95dd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:32.316468 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.316446 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-data" pod "961c8dee-4505-4b84-b984-3006b6a95dd8" (UID: "961c8dee-4505-4b84-b984-3006b6a95dd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:32.316603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.316485 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/961c8dee-4505-4b84-b984-3006b6a95dd8-istiod-ca-cert" pod "961c8dee-4505-4b84-b984-3006b6a95dd8" (UID: "961c8dee-4505-4b84-b984-3006b6a95dd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:32.316603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.316563 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-podinfo\") pod \"961c8dee-4505-4b84-b984-3006b6a95dd8\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " Oct 30 18:26:32.316722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.316615 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-token\" (UniqueName: \"kubernetes.io/projected/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-token\") pod \"961c8dee-4505-4b84-b984-3006b6a95dd8\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " Oct 30 18:26:32.316722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.316650 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/961c8dee-4505-4b84-b984-3006b6a95dd8-kube-api-access-dzb5v\" (UniqueName: \"kubernetes.io/projected/961c8dee-4505-4b84-b984-3006b6a95dd8-kube-api-access-dzb5v\") pod \"961c8dee-4505-4b84-b984-3006b6a95dd8\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " Oct 30 18:26:32.316825 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.316766 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-credential-socket\") pod \"961c8dee-4505-4b84-b984-3006b6a95dd8\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " Oct 30 18:26:32.316887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.316832 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-workload-socket\") pod \"961c8dee-4505-4b84-b984-3006b6a95dd8\" (UID: \"961c8dee-4505-4b84-b984-3006b6a95dd8\") " Oct 30 18:26:32.317367 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.317217 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-credential-socket" pod "961c8dee-4505-4b84-b984-3006b6a95dd8" (UID: "961c8dee-4505-4b84-b984-3006b6a95dd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:32.317484 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.317399 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-workload-socket" pod "961c8dee-4505-4b84-b984-3006b6a95dd8" (UID: "961c8dee-4505-4b84-b984-3006b6a95dd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:32.317897 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.317870 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:32.318007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.317902 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:32.318007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.317918 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:32.318007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.317931 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/961c8dee-4505-4b84-b984-3006b6a95dd8-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:32.318007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.317942 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:32.319951 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.319924 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-envoy" pod "961c8dee-4505-4b84-b984-3006b6a95dd8" (UID: "961c8dee-4505-4b84-b984-3006b6a95dd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:26:32.320271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.320249 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-podinfo" pod "961c8dee-4505-4b84-b984-3006b6a95dd8" (UID: "961c8dee-4505-4b84-b984-3006b6a95dd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:26:32.320357 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.320301 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961c8dee-4505-4b84-b984-3006b6a95dd8-kube-api-access-dzb5v" pod "961c8dee-4505-4b84-b984-3006b6a95dd8" (UID: "961c8dee-4505-4b84-b984-3006b6a95dd8"). InnerVolumeSpecName "kube-api-access-dzb5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:32.320450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.320433 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-token" pod "961c8dee-4505-4b84-b984-3006b6a95dd8" (UID: "961c8dee-4505-4b84-b984-3006b6a95dd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:32.419904 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.419870 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:32.419904 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.419899 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:32.420092 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.419912 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/961c8dee-4505-4b84-b984-3006b6a95dd8-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:32.420092 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.419925 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzb5v\" (UniqueName: \"kubernetes.io/projected/961c8dee-4505-4b84-b984-3006b6a95dd8-kube-api-access-dzb5v\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:32.601821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.601739 5232 generic.go:356] "Generic (PLEG): container finished" podID="961c8dee-4505-4b84-b984-3006b6a95dd8" containerID="53ff8a18a2e8a3e875aa6adcd6d5c9c11df3a5ea338f3151839b5b7e1ba28ebd" exitCode=0 Oct 30 18:26:32.601821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.601790 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" Oct 30 18:26:32.602363 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.601824 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" event={"ID":"961c8dee-4505-4b84-b984-3006b6a95dd8","Type":"ContainerDied","Data":"53ff8a18a2e8a3e875aa6adcd6d5c9c11df3a5ea338f3151839b5b7e1ba28ebd"} Oct 30 18:26:32.602363 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.601880 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x" event={"ID":"961c8dee-4505-4b84-b984-3006b6a95dd8","Type":"ContainerDied","Data":"bb83f5dce8a9b6e0f30708dc978a46039ae0f61c5fdb5bfb91db606a68e54453"} Oct 30 18:26:32.602363 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.601914 5232 scope.go:122] "RemoveContainer" containerID="53ff8a18a2e8a3e875aa6adcd6d5c9c11df3a5ea338f3151839b5b7e1ba28ebd" Oct 30 18:26:32.614409 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.614386 5232 scope.go:122] "RemoveContainer" containerID="615e3e5a46e56b54b6f6be9d6d6d7cccaa5dc27727a25838ad2b231bf448ab3e" Oct 30 18:26:32.626043 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.626024 5232 scope.go:122] "RemoveContainer" containerID="53ff8a18a2e8a3e875aa6adcd6d5c9c11df3a5ea338f3151839b5b7e1ba28ebd" Oct 30 18:26:32.626437 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:32.626410 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"53ff8a18a2e8a3e875aa6adcd6d5c9c11df3a5ea338f3151839b5b7e1ba28ebd\": container with ID starting with 53ff8a18a2e8a3e875aa6adcd6d5c9c11df3a5ea338f3151839b5b7e1ba28ebd not found: ID does not exist" containerID="53ff8a18a2e8a3e875aa6adcd6d5c9c11df3a5ea338f3151839b5b7e1ba28ebd" Oct 30 18:26:32.626546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.626446 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"53ff8a18a2e8a3e875aa6adcd6d5c9c11df3a5ea338f3151839b5b7e1ba28ebd"} err="failed to get container status \"53ff8a18a2e8a3e875aa6adcd6d5c9c11df3a5ea338f3151839b5b7e1ba28ebd\": rpc error: code = NotFound desc = could not find container \"53ff8a18a2e8a3e875aa6adcd6d5c9c11df3a5ea338f3151839b5b7e1ba28ebd\": container with ID starting with 53ff8a18a2e8a3e875aa6adcd6d5c9c11df3a5ea338f3151839b5b7e1ba28ebd not found: ID does not exist" Oct 30 18:26:32.626546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.626468 5232 scope.go:122] "RemoveContainer" containerID="615e3e5a46e56b54b6f6be9d6d6d7cccaa5dc27727a25838ad2b231bf448ab3e" Oct 30 18:26:32.626790 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:32.626769 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"615e3e5a46e56b54b6f6be9d6d6d7cccaa5dc27727a25838ad2b231bf448ab3e\": container with ID starting with 615e3e5a46e56b54b6f6be9d6d6d7cccaa5dc27727a25838ad2b231bf448ab3e not found: ID does not exist" containerID="615e3e5a46e56b54b6f6be9d6d6d7cccaa5dc27727a25838ad2b231bf448ab3e" Oct 30 18:26:32.626853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.626800 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"615e3e5a46e56b54b6f6be9d6d6d7cccaa5dc27727a25838ad2b231bf448ab3e"} err="failed to get container status \"615e3e5a46e56b54b6f6be9d6d6d7cccaa5dc27727a25838ad2b231bf448ab3e\": rpc error: code = NotFound desc = could not find container \"615e3e5a46e56b54b6f6be9d6d6d7cccaa5dc27727a25838ad2b231bf448ab3e\": container with ID starting with 615e3e5a46e56b54b6f6be9d6d6d7cccaa5dc27727a25838ad2b231bf448ab3e not found: ID does not exist" Oct 30 18:26:32.661058 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.661026 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x"] Oct 30 18:26:32.689850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.676400 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f8709f-openshift-default-5579d88b9-t8z6x"] Oct 30 18:26:32.745226 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.745200 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9" Oct 30 18:26:32.807433 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.807402 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="961c8dee-4505-4b84-b984-3006b6a95dd8" path="/var/lib/kubelet/pods/961c8dee-4505-4b84-b984-3006b6a95dd8/volumes" Oct 30 18:26:32.826912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.826885 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/36cd7371-04b2-45ad-ab88-88b3649868bb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36cd7371-04b2-45ad-ab88-88b3649868bb-maas-api-xks-ca\") pod \"36cd7371-04b2-45ad-ab88-88b3649868bb\" (UID: \"36cd7371-04b2-45ad-ab88-88b3649868bb\") " Oct 30 18:26:32.827088 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.826963 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/36cd7371-04b2-45ad-ab88-88b3649868bb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36cd7371-04b2-45ad-ab88-88b3649868bb-maas-api-openshift-ca\") pod \"36cd7371-04b2-45ad-ab88-88b3649868bb\" (UID: \"36cd7371-04b2-45ad-ab88-88b3649868bb\") " Oct 30 18:26:32.827088 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.827020 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/36cd7371-04b2-45ad-ab88-88b3649868bb-kube-api-access-ntrtg\" (UniqueName: \"kubernetes.io/projected/36cd7371-04b2-45ad-ab88-88b3649868bb-kube-api-access-ntrtg\") pod \"36cd7371-04b2-45ad-ab88-88b3649868bb\" (UID: \"36cd7371-04b2-45ad-ab88-88b3649868bb\") " Oct 30 18:26:32.827495 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.827380 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/36cd7371-04b2-45ad-ab88-88b3649868bb-maas-api-openshift-ca" pod "36cd7371-04b2-45ad-ab88-88b3649868bb" (UID: "36cd7371-04b2-45ad-ab88-88b3649868bb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:32.827787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.827766 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/36cd7371-04b2-45ad-ab88-88b3649868bb-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:32.829361 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.829335 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/36cd7371-04b2-45ad-ab88-88b3649868bb-maas-api-xks-ca" pod "36cd7371-04b2-45ad-ab88-88b3649868bb" (UID: "36cd7371-04b2-45ad-ab88-88b3649868bb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:26:32.829361 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.829341 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/36cd7371-04b2-45ad-ab88-88b3649868bb-kube-api-access-ntrtg" pod "36cd7371-04b2-45ad-ab88-88b3649868bb" (UID: "36cd7371-04b2-45ad-ab88-88b3649868bb"). InnerVolumeSpecName "kube-api-access-ntrtg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:32.933151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.933050 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntrtg\" (UniqueName: \"kubernetes.io/projected/36cd7371-04b2-45ad-ab88-88b3649868bb-kube-api-access-ntrtg\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:32.933151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:32.933093 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/36cd7371-04b2-45ad-ab88-88b3649868bb-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:33.622079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:33.618663 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9" Oct 30 18:26:33.622079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:33.619020 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9" event={"ID":"36cd7371-04b2-45ad-ab88-88b3649868bb","Type":"ContainerDied","Data":"d3b6cd0bcfafda2e6ae429653f4157f77adbb230f07239ceea9cc2a16534c3a3"} Oct 30 18:26:33.622079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:33.619079 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d3b6cd0bcfafda2e6ae429653f4157f77adbb230f07239ceea9cc2a16534c3a3" Oct 30 18:26:33.643539 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:33.643448 5232 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a994e2_22c0_49ce_8d6b_1898bf2f7edc.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod36cd7371_04b2_45ad_ab88_88b3649868bb.slice/crio-d3b6cd0bcfafda2e6ae429653f4157f77adbb230f07239ceea9cc2a16534c3a3\": RecentStats: unable to find data in memory cache]" Oct 30 18:26:33.893610 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:33.893525 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6"] Oct 30 18:26:33.898746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:33.898719 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-fb0abbf7-wj2n6"] Oct 30 18:26:34.805696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:34.805661 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0" path="/var/lib/kubelet/pods/bd43ca9d-4366-4bd2-80d4-8c63f9abbdd0/volumes" Oct 30 18:26:38.158387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:38.158358 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7"] Oct 30 18:26:38.193318 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:38.193286 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk"] Oct 30 18:26:39.662527 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:39.662470 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m"] Oct 30 18:26:39.693934 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:39.693905 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt"] Oct 30 18:26:40.643185 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:40.643157 5232 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29a994e2_22c0_49ce_8d6b_1898bf2f7edc.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:26:43.159067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.159030 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" podUID="365f97d1-14ff-488c-81aa-1d80e81d6de9" containerName="payload-processing" containerID="cri-o://f3f5eb6672930d3deb1bbb6f94bef96430ac7766049900e2aab8fb29a44f38ea" gracePeriod=25 Oct 30 18:26:43.194583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.194544 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" podUID="99d0f333-465d-4a99-b2af-a1208c0a12ef" containerName="payload-pre-processing" containerID="cri-o://0558ecf38a424601c31bd68cfff7c846cc1a68caf94efdaf3ef6ea188d05ccc9" gracePeriod=25 Oct 30 18:26:43.235863 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:26:43.235832 5232 logging.go:55] [core] [Channel #4030 SubChannel #4033]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.170:50052", ServerName: "10.133.0.170:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.170:50052: connect: connection refused" Oct 30 18:26:43.235863 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:26:43.235849 5232 logging.go:55] [core] [Channel #4031 SubChannel #4032]grpc: addrConn.createTransport failed to connect to {Addr: "10.133.0.171:50052", ServerName: "10.133.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.133.0.171:50052: connect: connection refused" Oct 30 18:26:43.558222 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.558189 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" Oct 30 18:26:43.581628 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.581607 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/365f97d1-14ff-488c-81aa-1d80e81d6de9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365f97d1-14ff-488c-81aa-1d80e81d6de9-plugins-config-volume\") pod \"365f97d1-14ff-488c-81aa-1d80e81d6de9\" (UID: \"365f97d1-14ff-488c-81aa-1d80e81d6de9\") " Oct 30 18:26:43.581706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.581658 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/365f97d1-14ff-488c-81aa-1d80e81d6de9-kube-api-access-8g84j\" (UniqueName: \"kubernetes.io/projected/365f97d1-14ff-488c-81aa-1d80e81d6de9-kube-api-access-8g84j\") pod \"365f97d1-14ff-488c-81aa-1d80e81d6de9\" (UID: \"365f97d1-14ff-488c-81aa-1d80e81d6de9\") " Oct 30 18:26:43.582138 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.582113 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/365f97d1-14ff-488c-81aa-1d80e81d6de9-plugins-config-volume" pod "365f97d1-14ff-488c-81aa-1d80e81d6de9" (UID: "365f97d1-14ff-488c-81aa-1d80e81d6de9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:43.583951 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.583929 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/365f97d1-14ff-488c-81aa-1d80e81d6de9-kube-api-access-8g84j" pod "365f97d1-14ff-488c-81aa-1d80e81d6de9" (UID: "365f97d1-14ff-488c-81aa-1d80e81d6de9"). InnerVolumeSpecName "kube-api-access-8g84j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:43.605316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.605289 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" Oct 30 18:26:43.684069 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.684039 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99d0f333-465d-4a99-b2af-a1208c0a12ef-kube-api-access-bllpj\" (UniqueName: \"kubernetes.io/projected/99d0f333-465d-4a99-b2af-a1208c0a12ef-kube-api-access-bllpj\") pod \"99d0f333-465d-4a99-b2af-a1208c0a12ef\" (UID: \"99d0f333-465d-4a99-b2af-a1208c0a12ef\") " Oct 30 18:26:43.684767 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.684359 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99d0f333-465d-4a99-b2af-a1208c0a12ef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d0f333-465d-4a99-b2af-a1208c0a12ef-plugins-config-volume\") pod \"99d0f333-465d-4a99-b2af-a1208c0a12ef\" (UID: \"99d0f333-465d-4a99-b2af-a1208c0a12ef\") " Oct 30 18:26:43.684991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.684746 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99d0f333-465d-4a99-b2af-a1208c0a12ef-plugins-config-volume" pod "99d0f333-465d-4a99-b2af-a1208c0a12ef" (UID: "99d0f333-465d-4a99-b2af-a1208c0a12ef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:43.685830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.685811 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99d0f333-465d-4a99-b2af-a1208c0a12ef-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:43.685980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.685968 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/365f97d1-14ff-488c-81aa-1d80e81d6de9-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:43.686064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.686054 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8g84j\" (UniqueName: \"kubernetes.io/projected/365f97d1-14ff-488c-81aa-1d80e81d6de9-kube-api-access-8g84j\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:43.686921 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.686892 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99d0f333-465d-4a99-b2af-a1208c0a12ef-kube-api-access-bllpj" pod "99d0f333-465d-4a99-b2af-a1208c0a12ef" (UID: "99d0f333-465d-4a99-b2af-a1208c0a12ef"). InnerVolumeSpecName "kube-api-access-bllpj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:43.738871 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.738841 5232 generic.go:356] "Generic (PLEG): container finished" podID="365f97d1-14ff-488c-81aa-1d80e81d6de9" containerID="f3f5eb6672930d3deb1bbb6f94bef96430ac7766049900e2aab8fb29a44f38ea" exitCode=0 Oct 30 18:26:43.739076 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.738897 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" Oct 30 18:26:43.739076 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.738937 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" event={"ID":"365f97d1-14ff-488c-81aa-1d80e81d6de9","Type":"ContainerDied","Data":"f3f5eb6672930d3deb1bbb6f94bef96430ac7766049900e2aab8fb29a44f38ea"} Oct 30 18:26:43.739076 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.738973 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" event={"ID":"365f97d1-14ff-488c-81aa-1d80e81d6de9","Type":"ContainerDied","Data":"fa3ba457261ce92283eabc016d583714e496a6c5025ede5c97d0ecf6014a964c"} Oct 30 18:26:43.739076 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.738995 5232 scope.go:122] "RemoveContainer" containerID="f3f5eb6672930d3deb1bbb6f94bef96430ac7766049900e2aab8fb29a44f38ea" Oct 30 18:26:43.741029 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.740949 5232 generic.go:356] "Generic (PLEG): container finished" podID="99d0f333-465d-4a99-b2af-a1208c0a12ef" containerID="0558ecf38a424601c31bd68cfff7c846cc1a68caf94efdaf3ef6ea188d05ccc9" exitCode=0 Oct 30 18:26:43.741029 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.740969 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" Oct 30 18:26:43.741282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.741114 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" event={"ID":"99d0f333-465d-4a99-b2af-a1208c0a12ef","Type":"ContainerDied","Data":"0558ecf38a424601c31bd68cfff7c846cc1a68caf94efdaf3ef6ea188d05ccc9"} Oct 30 18:26:43.741282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.741170 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" event={"ID":"99d0f333-465d-4a99-b2af-a1208c0a12ef","Type":"ContainerDied","Data":"649942b4f75a25ad6b5b921f27eb1a3e24fd72447f0f34f307afd322b5272f77"} Oct 30 18:26:43.751397 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.751246 5232 scope.go:122] "RemoveContainer" containerID="f3f5eb6672930d3deb1bbb6f94bef96430ac7766049900e2aab8fb29a44f38ea" Oct 30 18:26:43.751595 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:43.751575 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3f5eb6672930d3deb1bbb6f94bef96430ac7766049900e2aab8fb29a44f38ea\": container with ID starting with f3f5eb6672930d3deb1bbb6f94bef96430ac7766049900e2aab8fb29a44f38ea not found: ID does not exist" containerID="f3f5eb6672930d3deb1bbb6f94bef96430ac7766049900e2aab8fb29a44f38ea" Oct 30 18:26:43.751667 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.751604 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3f5eb6672930d3deb1bbb6f94bef96430ac7766049900e2aab8fb29a44f38ea"} err="failed to get container status \"f3f5eb6672930d3deb1bbb6f94bef96430ac7766049900e2aab8fb29a44f38ea\": rpc error: code = NotFound desc = could not find container \"f3f5eb6672930d3deb1bbb6f94bef96430ac7766049900e2aab8fb29a44f38ea\": container with ID starting with f3f5eb6672930d3deb1bbb6f94bef96430ac7766049900e2aab8fb29a44f38ea not found: ID does not exist" Oct 30 18:26:43.751667 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.751628 5232 scope.go:122] "RemoveContainer" containerID="0558ecf38a424601c31bd68cfff7c846cc1a68caf94efdaf3ef6ea188d05ccc9" Oct 30 18:26:43.762528 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.762490 5232 scope.go:122] "RemoveContainer" containerID="0558ecf38a424601c31bd68cfff7c846cc1a68caf94efdaf3ef6ea188d05ccc9" Oct 30 18:26:43.762856 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:43.762834 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0558ecf38a424601c31bd68cfff7c846cc1a68caf94efdaf3ef6ea188d05ccc9\": container with ID starting with 0558ecf38a424601c31bd68cfff7c846cc1a68caf94efdaf3ef6ea188d05ccc9 not found: ID does not exist" containerID="0558ecf38a424601c31bd68cfff7c846cc1a68caf94efdaf3ef6ea188d05ccc9" Oct 30 18:26:43.762909 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.762864 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0558ecf38a424601c31bd68cfff7c846cc1a68caf94efdaf3ef6ea188d05ccc9"} err="failed to get container status \"0558ecf38a424601c31bd68cfff7c846cc1a68caf94efdaf3ef6ea188d05ccc9\": rpc error: code = NotFound desc = could not find container \"0558ecf38a424601c31bd68cfff7c846cc1a68caf94efdaf3ef6ea188d05ccc9\": container with ID starting with 0558ecf38a424601c31bd68cfff7c846cc1a68caf94efdaf3ef6ea188d05ccc9 not found: ID does not exist" Oct 30 18:26:43.787303 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.787271 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bllpj\" (UniqueName: \"kubernetes.io/projected/99d0f333-465d-4a99-b2af-a1208c0a12ef-kube-api-access-bllpj\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:43.797947 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.797918 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk"] Oct 30 18:26:43.802301 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.802276 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk"] Oct 30 18:26:43.847161 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.847126 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7"] Oct 30 18:26:43.850816 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:43.850794 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7"] Oct 30 18:26:44.235979 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:44.235927 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-18cd5a-76676f5699-kfwwk" podUID="99d0f333-465d-4a99-b2af-a1208c0a12ef" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.171:50052\" within 1s: context deadline exceeded" Oct 30 18:26:44.236367 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:44.235936 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-api-b-18cd5a-57c7948996-9bgg7" podUID="365f97d1-14ff-488c-81aa-1d80e81d6de9" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.133.0.170:50052\" within 1s: context deadline exceeded" Oct 30 18:26:44.663828 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:44.663787 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m" podUID="4bb3aa87-34a9-42b3-ac1b-86919834f2d9" containerName="payload-processing" containerID="cri-o://3bbfd22855d3c49bc522f04f4ecf11cf93003189c246711e0d340df4ecdd7b6d" gracePeriod=25 Oct 30 18:26:44.694615 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:44.694581 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt" podUID="76395805-b5c2-4292-a77c-7dc14259a794" containerName="payload-pre-processing" containerID="cri-o://0d3373fc95ed2da292ecdd9e87338ac7cb5206ada5ad06b5f3c7d2ca32c29c37" gracePeriod=25 Oct 30 18:26:44.805995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:44.805956 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="365f97d1-14ff-488c-81aa-1d80e81d6de9" path="/var/lib/kubelet/pods/365f97d1-14ff-488c-81aa-1d80e81d6de9/volumes" Oct 30 18:26:44.806572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:44.806551 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99d0f333-465d-4a99-b2af-a1208c0a12ef" path="/var/lib/kubelet/pods/99d0f333-465d-4a99-b2af-a1208c0a12ef/volumes" Oct 30 18:26:45.061295 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.061269 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m" Oct 30 18:26:45.093192 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.093169 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt" Oct 30 18:26:45.109467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.109057 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4bb3aa87-34a9-42b3-ac1b-86919834f2d9-kube-api-access-2x5jh\" (UniqueName: \"kubernetes.io/projected/4bb3aa87-34a9-42b3-ac1b-86919834f2d9-kube-api-access-2x5jh\") pod \"4bb3aa87-34a9-42b3-ac1b-86919834f2d9\" (UID: \"4bb3aa87-34a9-42b3-ac1b-86919834f2d9\") " Oct 30 18:26:45.109467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.109353 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4bb3aa87-34a9-42b3-ac1b-86919834f2d9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb3aa87-34a9-42b3-ac1b-86919834f2d9-plugins-config-volume\") pod \"4bb3aa87-34a9-42b3-ac1b-86919834f2d9\" (UID: \"4bb3aa87-34a9-42b3-ac1b-86919834f2d9\") " Oct 30 18:26:45.130893 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.118796 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4bb3aa87-34a9-42b3-ac1b-86919834f2d9-plugins-config-volume" pod "4bb3aa87-34a9-42b3-ac1b-86919834f2d9" (UID: "4bb3aa87-34a9-42b3-ac1b-86919834f2d9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:45.142534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.142316 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4bb3aa87-34a9-42b3-ac1b-86919834f2d9-kube-api-access-2x5jh" pod "4bb3aa87-34a9-42b3-ac1b-86919834f2d9" (UID: "4bb3aa87-34a9-42b3-ac1b-86919834f2d9"). InnerVolumeSpecName "kube-api-access-2x5jh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:45.234608 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.226154 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76395805-b5c2-4292-a77c-7dc14259a794-kube-api-access-gs94t\" (UniqueName: \"kubernetes.io/projected/76395805-b5c2-4292-a77c-7dc14259a794-kube-api-access-gs94t\") pod \"76395805-b5c2-4292-a77c-7dc14259a794\" (UID: \"76395805-b5c2-4292-a77c-7dc14259a794\") " Oct 30 18:26:45.234608 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.226373 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/76395805-b5c2-4292-a77c-7dc14259a794-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76395805-b5c2-4292-a77c-7dc14259a794-plugins-config-volume\") pod \"76395805-b5c2-4292-a77c-7dc14259a794\" (UID: \"76395805-b5c2-4292-a77c-7dc14259a794\") " Oct 30 18:26:45.234608 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.226946 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4bb3aa87-34a9-42b3-ac1b-86919834f2d9-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:45.234608 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.226970 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2x5jh\" (UniqueName: \"kubernetes.io/projected/4bb3aa87-34a9-42b3-ac1b-86919834f2d9-kube-api-access-2x5jh\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:45.235065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.234754 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76395805-b5c2-4292-a77c-7dc14259a794-plugins-config-volume" pod "76395805-b5c2-4292-a77c-7dc14259a794" (UID: "76395805-b5c2-4292-a77c-7dc14259a794"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:26:45.241081 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.241021 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76395805-b5c2-4292-a77c-7dc14259a794-kube-api-access-gs94t" pod "76395805-b5c2-4292-a77c-7dc14259a794" (UID: "76395805-b5c2-4292-a77c-7dc14259a794"). InnerVolumeSpecName "kube-api-access-gs94t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:26:45.335389 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.335352 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gs94t\" (UniqueName: \"kubernetes.io/projected/76395805-b5c2-4292-a77c-7dc14259a794-kube-api-access-gs94t\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:45.335389 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.335384 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76395805-b5c2-4292-a77c-7dc14259a794-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:26:45.845926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.845887 5232 generic.go:356] "Generic (PLEG): container finished" podID="76395805-b5c2-4292-a77c-7dc14259a794" containerID="0d3373fc95ed2da292ecdd9e87338ac7cb5206ada5ad06b5f3c7d2ca32c29c37" exitCode=0 Oct 30 18:26:45.846278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.846249 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt" event={"ID":"76395805-b5c2-4292-a77c-7dc14259a794","Type":"ContainerDied","Data":"0d3373fc95ed2da292ecdd9e87338ac7cb5206ada5ad06b5f3c7d2ca32c29c37"} Oct 30 18:26:45.846489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.846473 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt" event={"ID":"76395805-b5c2-4292-a77c-7dc14259a794","Type":"ContainerDied","Data":"19a379f5623bc3057e54b5657e55b9557d5f03b6ba983f592d513cdaf8bd8c8b"} Oct 30 18:26:45.846661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.846645 5232 scope.go:122] "RemoveContainer" containerID="0d3373fc95ed2da292ecdd9e87338ac7cb5206ada5ad06b5f3c7d2ca32c29c37" Oct 30 18:26:45.847069 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.847049 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt" Oct 30 18:26:45.896370 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.896319 5232 generic.go:356] "Generic (PLEG): container finished" podID="4bb3aa87-34a9-42b3-ac1b-86919834f2d9" containerID="3bbfd22855d3c49bc522f04f4ecf11cf93003189c246711e0d340df4ecdd7b6d" exitCode=0 Oct 30 18:26:45.896753 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.896706 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m" event={"ID":"4bb3aa87-34a9-42b3-ac1b-86919834f2d9","Type":"ContainerDied","Data":"3bbfd22855d3c49bc522f04f4ecf11cf93003189c246711e0d340df4ecdd7b6d"} Oct 30 18:26:45.896949 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.896925 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m" event={"ID":"4bb3aa87-34a9-42b3-ac1b-86919834f2d9","Type":"ContainerDied","Data":"09bfc2338b860d049cf55ed708bc1a0e117be900268caefa6e155657def7788d"} Oct 30 18:26:45.897154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.897140 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m" Oct 30 18:26:45.992410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.992384 5232 scope.go:122] "RemoveContainer" containerID="0d3373fc95ed2da292ecdd9e87338ac7cb5206ada5ad06b5f3c7d2ca32c29c37" Oct 30 18:26:45.993714 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:45.993685 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d3373fc95ed2da292ecdd9e87338ac7cb5206ada5ad06b5f3c7d2ca32c29c37\": container with ID starting with 0d3373fc95ed2da292ecdd9e87338ac7cb5206ada5ad06b5f3c7d2ca32c29c37 not found: ID does not exist" containerID="0d3373fc95ed2da292ecdd9e87338ac7cb5206ada5ad06b5f3c7d2ca32c29c37" Oct 30 18:26:45.993844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.993728 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d3373fc95ed2da292ecdd9e87338ac7cb5206ada5ad06b5f3c7d2ca32c29c37"} err="failed to get container status \"0d3373fc95ed2da292ecdd9e87338ac7cb5206ada5ad06b5f3c7d2ca32c29c37\": rpc error: code = NotFound desc = could not find container \"0d3373fc95ed2da292ecdd9e87338ac7cb5206ada5ad06b5f3c7d2ca32c29c37\": container with ID starting with 0d3373fc95ed2da292ecdd9e87338ac7cb5206ada5ad06b5f3c7d2ca32c29c37 not found: ID does not exist" Oct 30 18:26:45.993844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.993753 5232 scope.go:122] "RemoveContainer" containerID="3bbfd22855d3c49bc522f04f4ecf11cf93003189c246711e0d340df4ecdd7b6d" Oct 30 18:26:45.996726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:45.996707 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt"] Oct 30 18:26:46.004381 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:46.004351 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-9066aa-cf84589ff-rppdt"] Oct 30 18:26:46.005184 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:46.005167 5232 scope.go:122] "RemoveContainer" containerID="3bbfd22855d3c49bc522f04f4ecf11cf93003189c246711e0d340df4ecdd7b6d" Oct 30 18:26:46.005467 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:46.005448 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3bbfd22855d3c49bc522f04f4ecf11cf93003189c246711e0d340df4ecdd7b6d\": container with ID starting with 3bbfd22855d3c49bc522f04f4ecf11cf93003189c246711e0d340df4ecdd7b6d not found: ID does not exist" containerID="3bbfd22855d3c49bc522f04f4ecf11cf93003189c246711e0d340df4ecdd7b6d" Oct 30 18:26:46.005596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:46.005486 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3bbfd22855d3c49bc522f04f4ecf11cf93003189c246711e0d340df4ecdd7b6d"} err="failed to get container status \"3bbfd22855d3c49bc522f04f4ecf11cf93003189c246711e0d340df4ecdd7b6d\": rpc error: code = NotFound desc = could not find container \"3bbfd22855d3c49bc522f04f4ecf11cf93003189c246711e0d340df4ecdd7b6d\": container with ID starting with 3bbfd22855d3c49bc522f04f4ecf11cf93003189c246711e0d340df4ecdd7b6d not found: ID does not exist" Oct 30 18:26:46.043419 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:46.043382 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m"] Oct 30 18:26:46.046967 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:46.046939 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-9066aa-c9bc596dd-l246m"] Oct 30 18:26:46.807902 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:46.807872 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4bb3aa87-34a9-42b3-ac1b-86919834f2d9" path="/var/lib/kubelet/pods/4bb3aa87-34a9-42b3-ac1b-86919834f2d9/volumes" Oct 30 18:26:46.808396 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:46.808378 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76395805-b5c2-4292-a77c-7dc14259a794" path="/var/lib/kubelet/pods/76395805-b5c2-4292-a77c-7dc14259a794/volumes" Oct 30 18:26:48.966685 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:48.966649 5232 patch_prober.go:29] interesting pod/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:26:48.967247 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:48.966715 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" podUID="7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:50.622654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.622616 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55"] Oct 30 18:26:50.624271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624229 5232 cpu_manager.go:414] "containerMap: removing container" podUID="c0d9b034-63e6-4493-94a5-43b41548d4ec" containerName="revoke-keys" Oct 30 18:26:50.624271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624261 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0d9b034-63e6-4493-94a5-43b41548d4ec" containerName="revoke-keys" Oct 30 18:26:50.624524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624280 5232 cpu_manager.go:414] "containerMap: removing container" podUID="36cd7371-04b2-45ad-ab88-88b3649868bb" containerName="revoke-keys" Oct 30 18:26:50.624524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624291 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="36cd7371-04b2-45ad-ab88-88b3649868bb" containerName="revoke-keys" Oct 30 18:26:50.624524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624301 5232 cpu_manager.go:414] "containerMap: removing container" podUID="365f97d1-14ff-488c-81aa-1d80e81d6de9" containerName="payload-processing" Oct 30 18:26:50.624524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624309 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="365f97d1-14ff-488c-81aa-1d80e81d6de9" containerName="payload-processing" Oct 30 18:26:50.624524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624329 5232 cpu_manager.go:414] "containerMap: removing container" podUID="76395805-b5c2-4292-a77c-7dc14259a794" containerName="payload-pre-processing" Oct 30 18:26:50.624524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624337 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76395805-b5c2-4292-a77c-7dc14259a794" containerName="payload-pre-processing" Oct 30 18:26:50.624524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624379 5232 cpu_manager.go:414] "containerMap: removing container" podUID="4bb3aa87-34a9-42b3-ac1b-86919834f2d9" containerName="payload-processing" Oct 30 18:26:50.624524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624387 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4bb3aa87-34a9-42b3-ac1b-86919834f2d9" containerName="payload-processing" Oct 30 18:26:50.624524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624400 5232 cpu_manager.go:414] "containerMap: removing container" podUID="99d0f333-465d-4a99-b2af-a1208c0a12ef" containerName="payload-pre-processing" Oct 30 18:26:50.624524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624408 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99d0f333-465d-4a99-b2af-a1208c0a12ef" containerName="payload-pre-processing" Oct 30 18:26:50.624524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624420 5232 cpu_manager.go:414] "containerMap: removing container" podUID="961c8dee-4505-4b84-b984-3006b6a95dd8" containerName="istio-proxy" Oct 30 18:26:50.624524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624427 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961c8dee-4505-4b84-b984-3006b6a95dd8" containerName="istio-proxy" Oct 30 18:26:50.624524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624460 5232 cpu_manager.go:414] "containerMap: removing container" podUID="961c8dee-4505-4b84-b984-3006b6a95dd8" containerName="istio-proxy" Oct 30 18:26:50.624524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624468 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="961c8dee-4505-4b84-b984-3006b6a95dd8" containerName="istio-proxy" Oct 30 18:26:50.625342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624647 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="36cd7371-04b2-45ad-ab88-88b3649868bb" containerName="revoke-keys" Oct 30 18:26:50.625342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624666 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="365f97d1-14ff-488c-81aa-1d80e81d6de9" containerName="payload-processing" Oct 30 18:26:50.625342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624680 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="76395805-b5c2-4292-a77c-7dc14259a794" containerName="payload-pre-processing" Oct 30 18:26:50.625342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624696 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="4bb3aa87-34a9-42b3-ac1b-86919834f2d9" containerName="payload-processing" Oct 30 18:26:50.625342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624708 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="99d0f333-465d-4a99-b2af-a1208c0a12ef" containerName="payload-pre-processing" Oct 30 18:26:50.625342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624718 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="961c8dee-4505-4b84-b984-3006b6a95dd8" containerName="istio-proxy" Oct 30 18:26:50.625342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624731 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="961c8dee-4505-4b84-b984-3006b6a95dd8" containerName="istio-proxy" Oct 30 18:26:50.625342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.624744 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0d9b034-63e6-4493-94a5-43b41548d4ec" containerName="revoke-keys" Oct 30 18:26:50.629351 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.629331 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.657588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.657481 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-0f0c50-openshift-default-dockercfg-xj27w\"" Oct 30 18:26:50.663417 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.663292 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55"] Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.840988 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-data\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.841045 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-token\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.841109 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istiod-ca-cert\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.841266 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-credential-socket\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.841300 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-envoy\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.841374 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-workload-socket\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.841418 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-workload-certs\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.841444 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v4tnc\" (UniqueName: \"kubernetes.io/projected/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-kube-api-access-v4tnc\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.841475 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-podinfo\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.942613 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-credential-socket\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.942665 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-envoy\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.942741 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-workload-socket\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.942805 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-workload-certs\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.942831 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v4tnc\" (UniqueName: \"kubernetes.io/projected/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-kube-api-access-v4tnc\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.942862 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-podinfo\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.942963 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-data\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.942998 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-token\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.943060 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istiod-ca-cert\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.944037 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istiod-ca-cert\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.944448 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-credential-socket\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.945135 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-workload-certs\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.945575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.945397 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-workload-socket\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.947028 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.945795 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-data\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.989543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.984798 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-envoy\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.997549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.994111 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-podinfo\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.997549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.995616 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v4tnc\" (UniqueName: \"kubernetes.io/projected/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-kube-api-access-v4tnc\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:50.997549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:50.996078 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-token\") pod \"e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:51.256534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:51.256135 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:52.358604 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:52.358564 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm"] Oct 30 18:26:52.392806 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:52.392756 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:26:52.393058 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:52.393025 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:26:52.393269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:52.393231 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-295497-h8kfm"] Oct 30 18:26:52.393467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:52.393441 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:26:52.400978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:52.399585 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55"] Oct 30 18:26:52.903803 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:52.903705 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c9da94e-bcbf-4490-89f4-cce4c3505f88" path="/var/lib/kubelet/pods/1c9da94e-bcbf-4490-89f4-cce4c3505f88/volumes" Oct 30 18:26:53.251595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:53.238625 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-wmxgt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:26:53.251595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:53.238689 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-wmxgt" podUID="4071cbed-2e82-4a41-97a0-ec2a30675d06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:53.454916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:53.454833 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cxtkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:26:53.455389 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:53.455355 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" podUID="0ea67986-73d8-4a9d-9268-a664b61c099c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.23:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:26:53.455552 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:53.455399 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" event={"ID":"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac","Type":"ContainerStarted","Data":"78057977b8c711a6e9d6bdf34ef949a035e90cf4113d1777a0d5f0e401cd9865"} Oct 30 18:26:53.455748 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:53.455726 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" event={"ID":"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac","Type":"ContainerStarted","Data":"cc6e9d144a7dbd27bdaea73425ddb73d201451a3b7b430be228993ae78c157f3"} Oct 30 18:26:53.509960 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:53.509841 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" podStartSLOduration=3.509822136 podStartE2EDuration="3.509822136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:53.504780534 +0000 UTC m=+1512.821200855" watchObservedRunningTime="2026-10-30 18:26:53.509822136 +0000 UTC m=+1512.826242459" Oct 30 18:26:54.257352 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:54.257321 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:54.263667 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:54.263640 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:54.366031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:54.365997 5232 scope.go:122] "RemoveContainer" containerID="2e57e08fb7173869653c5c5052b9592bdd179ec9b6aa7be040cf433bfd2dedba" Oct 30 18:26:54.376385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:54.376361 5232 scope.go:122] "RemoveContainer" containerID="83f572f64a38a5d19b391a47d8eb61e8a38202a8df7ea0e641cfd804008c5747" Oct 30 18:26:54.465721 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:54.465692 5232 generic.go:356] "Generic (PLEG): container finished" podID="7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" containerID="760b4a2fa1a6e8045933313a9b9c2113acbddd616d424c879f758c55680d476e" exitCode=0 Oct 30 18:26:54.466049 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:54.465762 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" event={"ID":"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6","Type":"ContainerDied","Data":"760b4a2fa1a6e8045933313a9b9c2113acbddd616d424c879f758c55680d476e"} Oct 30 18:26:54.466103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:54.466063 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:54.466498 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:54.466482 5232 scope.go:122] "RemoveContainer" containerID="760b4a2fa1a6e8045933313a9b9c2113acbddd616d424c879f758c55680d476e" Oct 30 18:26:54.467051 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:54.467030 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:26:54.472702 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:54.469878 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:26:54.472702 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:54.470012 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:26:54.472702 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:54.470064 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:26:55.475265 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:55.475226 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" event={"ID":"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6","Type":"ContainerStarted","Data":"e298fbbad008039389123aaf40766a83939a57e825111f4254a3ae258f2b3cf2"} Oct 30 18:26:55.976949 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:55.976912 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:26:55.977164 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:55.976966 5232 patch_prober.go:29] interesting pod/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 18:26:55.977164 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:55.977027 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" podUID="7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 18:26:56.974075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:56.974038 5232 patch_prober.go:29] interesting pod/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 18:26:56.974491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:56.974094 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" podUID="7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 18:26:57.060082 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:26:57.060045 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:26:57.892396 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:57.892360 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7"] Oct 30 18:26:57.897428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:57.897402 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" Oct 30 18:26:57.950471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:57.949328 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-0f0c50\"" Oct 30 18:26:57.950471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:57.949462 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-0f0c50-dockercfg-nvqg5\"" Oct 30 18:26:57.950471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:57.950364 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7"] Oct 30 18:26:57.975091 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:57.975044 5232 patch_prober.go:29] interesting pod/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 18:26:58.236724 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.236685 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" podUID="7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 18:26:58.237307 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.067474 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29af1a00-c1ac-401e-abcd-eefeaafab0de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7\" (UID: \"29af1a00-c1ac-401e-abcd-eefeaafab0de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" Oct 30 18:26:58.237777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.237661 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qz6fn\" (UniqueName: \"kubernetes.io/projected/29af1a00-c1ac-401e-abcd-eefeaafab0de-kube-api-access-qz6fn\") pod \"payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7\" (UID: \"29af1a00-c1ac-401e-abcd-eefeaafab0de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" Oct 30 18:26:58.318629 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.318593 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz"] Oct 30 18:26:58.326568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.324901 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" Oct 30 18:26:58.341941 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.341909 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz"] Oct 30 18:26:58.344619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.342714 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qz6fn\" (UniqueName: \"kubernetes.io/projected/29af1a00-c1ac-401e-abcd-eefeaafab0de-kube-api-access-qz6fn\") pod \"payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7\" (UID: \"29af1a00-c1ac-401e-abcd-eefeaafab0de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" Oct 30 18:26:58.344619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.344257 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29af1a00-c1ac-401e-abcd-eefeaafab0de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7\" (UID: \"29af1a00-c1ac-401e-abcd-eefeaafab0de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" Oct 30 18:26:58.345227 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.345204 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29af1a00-c1ac-401e-abcd-eefeaafab0de-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7\" (UID: \"29af1a00-c1ac-401e-abcd-eefeaafab0de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" Oct 30 18:26:58.362125 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.362101 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qz6fn\" (UniqueName: \"kubernetes.io/projected/29af1a00-c1ac-401e-abcd-eefeaafab0de-kube-api-access-qz6fn\") pod \"payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7\" (UID: \"29af1a00-c1ac-401e-abcd-eefeaafab0de\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" Oct 30 18:26:58.445567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.445496 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz\" (UID: \"f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" Oct 30 18:26:58.445739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.445615 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hxrvx\" (UniqueName: \"kubernetes.io/projected/f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b-kube-api-access-hxrvx\") pod \"payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz\" (UID: \"f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" Oct 30 18:26:58.547555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.547446 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz\" (UID: \"f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" Oct 30 18:26:58.547555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.547496 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hxrvx\" (UniqueName: \"kubernetes.io/projected/f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b-kube-api-access-hxrvx\") pod \"payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz\" (UID: \"f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" Oct 30 18:26:58.548339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.548308 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz\" (UID: \"f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" Oct 30 18:26:58.559115 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.559092 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" Oct 30 18:26:58.568888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.568867 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hxrvx\" (UniqueName: \"kubernetes.io/projected/f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b-kube-api-access-hxrvx\") pod \"payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz\" (UID: \"f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b\") " pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" Oct 30 18:26:58.651400 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.651370 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" Oct 30 18:26:58.970004 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:26:58.969967 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod29af1a00_c1ac_401e_abcd_eefeaafab0de.slice/crio-5e79ee205835b2e027b1722b2583148dd54e978d047a7f99f3dfe89fe7af5a26 WatchSource:0}: Error finding container 5e79ee205835b2e027b1722b2583148dd54e978d047a7f99f3dfe89fe7af5a26: Status 404 returned error can't find the container with id 5e79ee205835b2e027b1722b2583148dd54e978d047a7f99f3dfe89fe7af5a26 Oct 30 18:26:58.971087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.971013 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7"] Oct 30 18:26:58.974292 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.974268 5232 patch_prober.go:29] interesting pod/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 18:26:58.974464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:58.974439 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" podUID="7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 18:26:59.240420 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:26:59.240378 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf83cfef5_1eeb_4d0f_9cc8_bb9de0d38d8b.slice/crio-5b2c3c61b7facbcbd87c1084208f337768f5685c87ef307d5c1af5d7576984a6 WatchSource:0}: Error finding container 5b2c3c61b7facbcbd87c1084208f337768f5685c87ef307d5c1af5d7576984a6: Status 404 returned error can't find the container with id 5b2c3c61b7facbcbd87c1084208f337768f5685c87ef307d5c1af5d7576984a6 Oct 30 18:26:59.241770 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:59.241746 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz"] Oct 30 18:26:59.583536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:59.573131 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" event={"ID":"f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b","Type":"ContainerStarted","Data":"cae053a99026cc0ed6be477cb7ece43788a46b1187c436d947539cdfee1e33e1"} Oct 30 18:26:59.583536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:59.573199 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" event={"ID":"f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b","Type":"ContainerStarted","Data":"5b2c3c61b7facbcbd87c1084208f337768f5685c87ef307d5c1af5d7576984a6"} Oct 30 18:26:59.583536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:59.574432 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" Oct 30 18:26:59.583536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:59.576802 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" event={"ID":"29af1a00-c1ac-401e-abcd-eefeaafab0de","Type":"ContainerStarted","Data":"336083d06a070e03fd45ca69dea9d416af352630236c85cf0127d784e060d077"} Oct 30 18:26:59.583536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:59.576843 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" event={"ID":"29af1a00-c1ac-401e-abcd-eefeaafab0de","Type":"ContainerStarted","Data":"5e79ee205835b2e027b1722b2583148dd54e978d047a7f99f3dfe89fe7af5a26"} Oct 30 18:26:59.594688 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:59.591415 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" Oct 30 18:26:59.723575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:59.722966 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" podStartSLOduration=2.72294266 podStartE2EDuration="2.72294266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:59.65196716 +0000 UTC m=+1518.968387483" watchObservedRunningTime="2026-10-30 18:26:59.72294266 +0000 UTC m=+1519.039362984" Oct 30 18:26:59.988601 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:59.988358 5232 patch_prober.go:29] interesting pod/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 18:26:59.989047 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:26:59.988496 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" podUID="7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 18:27:00.054898 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.054848 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" podStartSLOduration=3.054833491 podStartE2EDuration="3.054833491s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:26:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:26:59.71879836 +0000 UTC m=+1519.035218693" watchObservedRunningTime="2026-10-30 18:27:00.054833491 +0000 UTC m=+1519.371253817" Oct 30 18:27:00.057493 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.057461 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt"] Oct 30 18:27:00.062637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.062620 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt" Oct 30 18:27:00.080873 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.080841 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt"] Oct 30 18:27:00.089216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.089187 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:27:00.089572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.089552 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:27:00.239785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.239690 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xg4fw\" (UniqueName: \"kubernetes.io/projected/2ac3d81c-13cd-4273-b143-7b78579b24cb-kube-api-access-xg4fw\") pod \"maas-api-revoke-keys-e2e-api-b-550276-k26kt\" (UID: \"2ac3d81c-13cd-4273-b143-7b78579b24cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt" Oct 30 18:27:00.239959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.239891 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac3d81c-13cd-4273-b143-7b78579b24cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-550276-k26kt\" (UID: \"2ac3d81c-13cd-4273-b143-7b78579b24cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt" Oct 30 18:27:00.240058 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.240037 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac3d81c-13cd-4273-b143-7b78579b24cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-550276-k26kt\" (UID: \"2ac3d81c-13cd-4273-b143-7b78579b24cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt" Oct 30 18:27:00.341394 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.341361 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xg4fw\" (UniqueName: \"kubernetes.io/projected/2ac3d81c-13cd-4273-b143-7b78579b24cb-kube-api-access-xg4fw\") pod \"maas-api-revoke-keys-e2e-api-b-550276-k26kt\" (UID: \"2ac3d81c-13cd-4273-b143-7b78579b24cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt" Oct 30 18:27:00.341835 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.341485 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac3d81c-13cd-4273-b143-7b78579b24cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-550276-k26kt\" (UID: \"2ac3d81c-13cd-4273-b143-7b78579b24cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt" Oct 30 18:27:00.341835 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.341597 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac3d81c-13cd-4273-b143-7b78579b24cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-550276-k26kt\" (UID: \"2ac3d81c-13cd-4273-b143-7b78579b24cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt" Oct 30 18:27:00.342289 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.342263 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac3d81c-13cd-4273-b143-7b78579b24cb-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-550276-k26kt\" (UID: \"2ac3d81c-13cd-4273-b143-7b78579b24cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt" Oct 30 18:27:00.344123 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.344093 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac3d81c-13cd-4273-b143-7b78579b24cb-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-550276-k26kt\" (UID: \"2ac3d81c-13cd-4273-b143-7b78579b24cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt" Oct 30 18:27:00.360811 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.360783 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xg4fw\" (UniqueName: \"kubernetes.io/projected/2ac3d81c-13cd-4273-b143-7b78579b24cb-kube-api-access-xg4fw\") pod \"maas-api-revoke-keys-e2e-api-b-550276-k26kt\" (UID: \"2ac3d81c-13cd-4273-b143-7b78579b24cb\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt" Oct 30 18:27:00.379502 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.379481 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt" Oct 30 18:27:00.738062 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.738017 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt"] Oct 30 18:27:00.739753 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:27:00.739707 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac3d81c_13cd_4273_b143_7b78579b24cb.slice/crio-000976e75d378282ad044ef8bfed9a93a6513cca9487f22b09bd79055ad91eee WatchSource:0}: Error finding container 000976e75d378282ad044ef8bfed9a93a6513cca9487f22b09bd79055ad91eee: Status 404 returned error can't find the container with id 000976e75d378282ad044ef8bfed9a93a6513cca9487f22b09bd79055ad91eee Oct 30 18:27:00.979534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.977908 5232 patch_prober.go:29] interesting pod/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" start-of-body= Oct 30 18:27:00.979534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:00.977972 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" podUID="7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.166:15021/healthz/ready\": dial tcp 10.133.0.166:15021: connect: connection refused" Oct 30 18:27:01.597688 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:01.597649 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt" event={"ID":"2ac3d81c-13cd-4273-b143-7b78579b24cb","Type":"ContainerStarted","Data":"3901ec1a1662a958218176f85de80b23a41c000a79110836fb227138a5e791d3"} Oct 30 18:27:01.598107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:01.597714 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt" event={"ID":"2ac3d81c-13cd-4273-b143-7b78579b24cb","Type":"ContainerStarted","Data":"000976e75d378282ad044ef8bfed9a93a6513cca9487f22b09bd79055ad91eee"} Oct 30 18:27:01.641058 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:01.640996 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt" podStartSLOduration=1.6409765950000001 podStartE2EDuration="1.640976595s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:01.638445685 +0000 UTC m=+1520.954866000" watchObservedRunningTime="2026-10-30 18:27:01.640976595 +0000 UTC m=+1520.957396918" Oct 30 18:27:01.979656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:01.979585 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:27:01.980075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:01.980056 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:27:01.980864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:01.980847 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:27:02.608718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:02.608674 5232 generic.go:356] "Generic (PLEG): container finished" podID="2ac3d81c-13cd-4273-b143-7b78579b24cb" containerID="3901ec1a1662a958218176f85de80b23a41c000a79110836fb227138a5e791d3" exitCode=0 Oct 30 18:27:02.609210 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:02.608747 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt" event={"ID":"2ac3d81c-13cd-4273-b143-7b78579b24cb","Type":"ContainerDied","Data":"3901ec1a1662a958218176f85de80b23a41c000a79110836fb227138a5e791d3"} Oct 30 18:27:03.775111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:03.775085 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt" Oct 30 18:27:03.909982 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:03.909900 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac3d81c-13cd-4273-b143-7b78579b24cb-kube-api-access-xg4fw\" (UniqueName: \"kubernetes.io/projected/2ac3d81c-13cd-4273-b143-7b78579b24cb-kube-api-access-xg4fw\") pod \"2ac3d81c-13cd-4273-b143-7b78579b24cb\" (UID: \"2ac3d81c-13cd-4273-b143-7b78579b24cb\") " Oct 30 18:27:03.910153 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:03.910103 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ac3d81c-13cd-4273-b143-7b78579b24cb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac3d81c-13cd-4273-b143-7b78579b24cb-maas-api-xks-ca\") pod \"2ac3d81c-13cd-4273-b143-7b78579b24cb\" (UID: \"2ac3d81c-13cd-4273-b143-7b78579b24cb\") " Oct 30 18:27:03.910245 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:03.910228 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ac3d81c-13cd-4273-b143-7b78579b24cb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac3d81c-13cd-4273-b143-7b78579b24cb-maas-api-openshift-ca\") pod \"2ac3d81c-13cd-4273-b143-7b78579b24cb\" (UID: \"2ac3d81c-13cd-4273-b143-7b78579b24cb\") " Oct 30 18:27:03.910575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:03.910548 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac3d81c-13cd-4273-b143-7b78579b24cb-maas-api-openshift-ca" pod "2ac3d81c-13cd-4273-b143-7b78579b24cb" (UID: "2ac3d81c-13cd-4273-b143-7b78579b24cb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:03.910787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:03.910774 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac3d81c-13cd-4273-b143-7b78579b24cb-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:03.912173 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:03.912151 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac3d81c-13cd-4273-b143-7b78579b24cb-kube-api-access-xg4fw" pod "2ac3d81c-13cd-4273-b143-7b78579b24cb" (UID: "2ac3d81c-13cd-4273-b143-7b78579b24cb"). InnerVolumeSpecName "kube-api-access-xg4fw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:03.912227 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:03.912191 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac3d81c-13cd-4273-b143-7b78579b24cb-maas-api-xks-ca" pod "2ac3d81c-13cd-4273-b143-7b78579b24cb" (UID: "2ac3d81c-13cd-4273-b143-7b78579b24cb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:04.013613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:04.012341 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xg4fw\" (UniqueName: \"kubernetes.io/projected/2ac3d81c-13cd-4273-b143-7b78579b24cb-kube-api-access-xg4fw\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:04.013613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:04.012376 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac3d81c-13cd-4273-b143-7b78579b24cb-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:04.627298 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:04.627275 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt" Oct 30 18:27:04.627628 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:04.627302 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt" event={"ID":"2ac3d81c-13cd-4273-b143-7b78579b24cb","Type":"ContainerDied","Data":"000976e75d378282ad044ef8bfed9a93a6513cca9487f22b09bd79055ad91eee"} Oct 30 18:27:04.627628 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:04.627377 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="000976e75d378282ad044ef8bfed9a93a6513cca9487f22b09bd79055ad91eee" Oct 30 18:27:11.600754 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:11.600722 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" Oct 30 18:27:11.601224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:11.600791 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" Oct 30 18:27:18.108723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:18.108688 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9"] Oct 30 18:27:18.120766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:18.120733 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-9066aa-6nmk9"] Oct 30 18:27:18.807635 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:18.807596 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="36cd7371-04b2-45ad-ab88-88b3649868bb" path="/var/lib/kubelet/pods/36cd7371-04b2-45ad-ab88-88b3649868bb/volumes" Oct 30 18:27:23.964886 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:23.964805 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft"] Oct 30 18:27:23.965373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:23.965088 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" podUID="368378dd-a0ab-49f6-8c9d-8f457863b3ec" containerName="istio-proxy" containerID="cri-o://48b4a7fe8f0e88fd9fd2b05045a4e31c6b46164bff3895646f5261e88a6afb62" gracePeriod=30 Oct 30 18:27:25.617247 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:25.617213 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886"] Oct 30 18:27:25.628986 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:25.628959 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-18cd5a-6h886"] Oct 30 18:27:26.065582 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.064828 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2"] Oct 30 18:27:26.065582 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.065107 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" podUID="7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" containerName="istio-proxy" containerID="cri-o://e298fbbad008039389123aaf40766a83939a57e825111f4254a3ae258f2b3cf2" gracePeriod=30 Oct 30 18:27:26.702080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.702046 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28"] Oct 30 18:27:26.702805 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.702788 5232 cpu_manager.go:414] "containerMap: removing container" podUID="2ac3d81c-13cd-4273-b143-7b78579b24cb" containerName="revoke-keys" Oct 30 18:27:26.702805 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.702806 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ac3d81c-13cd-4273-b143-7b78579b24cb" containerName="revoke-keys" Oct 30 18:27:26.702959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.702948 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ac3d81c-13cd-4273-b143-7b78579b24cb" containerName="revoke-keys" Oct 30 18:27:26.708429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.708410 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28" Oct 30 18:27:26.723348 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.723325 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:27:26.726219 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.726194 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28"] Oct 30 18:27:26.732730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.732709 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:27:26.804146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.804115 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0d9b034-63e6-4493-94a5-43b41548d4ec" path="/var/lib/kubelet/pods/c0d9b034-63e6-4493-94a5-43b41548d4ec/volumes" Oct 30 18:27:26.815419 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.815392 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0963e410-ad61-4bae-b699-997384481b61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-227c01-b9v28\" (UID: \"0963e410-ad61-4bae-b699-997384481b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28" Oct 30 18:27:26.815576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.815523 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fb846\" (UniqueName: \"kubernetes.io/projected/0963e410-ad61-4bae-b699-997384481b61-kube-api-access-fb846\") pod \"maas-api-revoke-keys-e2e-api-b-227c01-b9v28\" (UID: \"0963e410-ad61-4bae-b699-997384481b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28" Oct 30 18:27:26.815628 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.815615 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0963e410-ad61-4bae-b699-997384481b61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-227c01-b9v28\" (UID: \"0963e410-ad61-4bae-b699-997384481b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28" Oct 30 18:27:26.917402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.917370 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0963e410-ad61-4bae-b699-997384481b61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-227c01-b9v28\" (UID: \"0963e410-ad61-4bae-b699-997384481b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28" Oct 30 18:27:26.917578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.917567 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0963e410-ad61-4bae-b699-997384481b61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-227c01-b9v28\" (UID: \"0963e410-ad61-4bae-b699-997384481b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28" Oct 30 18:27:26.917644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.917622 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fb846\" (UniqueName: \"kubernetes.io/projected/0963e410-ad61-4bae-b699-997384481b61-kube-api-access-fb846\") pod \"maas-api-revoke-keys-e2e-api-b-227c01-b9v28\" (UID: \"0963e410-ad61-4bae-b699-997384481b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28" Oct 30 18:27:26.918178 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.918157 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0963e410-ad61-4bae-b699-997384481b61-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-227c01-b9v28\" (UID: \"0963e410-ad61-4bae-b699-997384481b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28" Oct 30 18:27:26.919889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.919867 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0963e410-ad61-4bae-b699-997384481b61-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-227c01-b9v28\" (UID: \"0963e410-ad61-4bae-b699-997384481b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28" Oct 30 18:27:26.939832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:26.939810 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fb846\" (UniqueName: \"kubernetes.io/projected/0963e410-ad61-4bae-b699-997384481b61-kube-api-access-fb846\") pod \"maas-api-revoke-keys-e2e-api-b-227c01-b9v28\" (UID: \"0963e410-ad61-4bae-b699-997384481b61\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28" Oct 30 18:27:27.027342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:27.027310 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28" Oct 30 18:27:27.037876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:27.037840 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc"] Oct 30 18:27:27.059881 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:27.059840 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7cbf63e5-8jnvc"] Oct 30 18:27:27.061381 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:27:27.061337 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:27:27.357544 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:27:27.357497 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0963e410_ad61_4bae_b699_997384481b61.slice/crio-7116b2c8b8f3e7b728bb2f692c2b95f7833272079e5f9ab7dd9877a76844058c WatchSource:0}: Error finding container 7116b2c8b8f3e7b728bb2f692c2b95f7833272079e5f9ab7dd9877a76844058c: Status 404 returned error can't find the container with id 7116b2c8b8f3e7b728bb2f692c2b95f7833272079e5f9ab7dd9877a76844058c Oct 30 18:27:27.359336 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:27.359309 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28"] Oct 30 18:27:28.010180 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:28.010146 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28" event={"ID":"0963e410-ad61-4bae-b699-997384481b61","Type":"ContainerStarted","Data":"5959ab3b577a1234a4b72d519e0309c13ba3e940ec36ec714e0eb7f73211ccd8"} Oct 30 18:27:28.010636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:28.010202 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28" event={"ID":"0963e410-ad61-4bae-b699-997384481b61","Type":"ContainerStarted","Data":"7116b2c8b8f3e7b728bb2f692c2b95f7833272079e5f9ab7dd9877a76844058c"} Oct 30 18:27:28.053559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:28.053482 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28" podStartSLOduration=2.053468044 podStartE2EDuration="2.053468044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:28.050665024 +0000 UTC m=+1547.367085350" watchObservedRunningTime="2026-10-30 18:27:28.053468044 +0000 UTC m=+1547.369888364" Oct 30 18:27:28.805129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:28.805099 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="79cb5f7a-0cbb-420f-886b-cd55e96f16ee" path="/var/lib/kubelet/pods/79cb5f7a-0cbb-420f-886b-cd55e96f16ee/volumes" Oct 30 18:27:29.019243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.019218 5232 generic.go:356] "Generic (PLEG): container finished" podID="0963e410-ad61-4bae-b699-997384481b61" containerID="5959ab3b577a1234a4b72d519e0309c13ba3e940ec36ec714e0eb7f73211ccd8" exitCode=0 Oct 30 18:27:29.019610 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.019275 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28" event={"ID":"0963e410-ad61-4bae-b699-997384481b61","Type":"ContainerDied","Data":"5959ab3b577a1234a4b72d519e0309c13ba3e940ec36ec714e0eb7f73211ccd8"} Oct 30 18:27:29.451176 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.451149 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:27:29.586769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.586738 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-envoy\") pod \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " Oct 30 18:27:29.586961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.586783 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-workload-certs\") pod \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " Oct 30 18:27:29.586961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.586880 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-credential-socket\") pod \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " Oct 30 18:27:29.586961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.586905 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/368378dd-a0ab-49f6-8c9d-8f457863b3ec-kube-api-access-qqtzg\" (UniqueName: \"kubernetes.io/projected/368378dd-a0ab-49f6-8c9d-8f457863b3ec-kube-api-access-qqtzg\") pod \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " Oct 30 18:27:29.586961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.586935 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-data\") pod \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " Oct 30 18:27:29.586961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.586961 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-podinfo\") pod \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " Oct 30 18:27:29.587270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.587066 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-token\" (UniqueName: \"kubernetes.io/projected/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-token\") pod \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " Oct 30 18:27:29.587270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.587137 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-workload-socket\") pod \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " Oct 30 18:27:29.587270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.587135 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-workload-certs" pod "368378dd-a0ab-49f6-8c9d-8f457863b3ec" (UID: "368378dd-a0ab-49f6-8c9d-8f457863b3ec"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:29.587270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.587193 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-credential-socket" pod "368378dd-a0ab-49f6-8c9d-8f457863b3ec" (UID: "368378dd-a0ab-49f6-8c9d-8f457863b3ec"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:29.587270 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.587208 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istiod-ca-cert\") pod \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\" (UID: \"368378dd-a0ab-49f6-8c9d-8f457863b3ec\") " Oct 30 18:27:29.587795 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.587314 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-data" pod "368378dd-a0ab-49f6-8c9d-8f457863b3ec" (UID: "368378dd-a0ab-49f6-8c9d-8f457863b3ec"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:29.587795 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.587410 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-workload-socket" pod "368378dd-a0ab-49f6-8c9d-8f457863b3ec" (UID: "368378dd-a0ab-49f6-8c9d-8f457863b3ec"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:29.587920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.587894 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istiod-ca-cert" pod "368378dd-a0ab-49f6-8c9d-8f457863b3ec" (UID: "368378dd-a0ab-49f6-8c9d-8f457863b3ec"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:29.588198 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.588170 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:29.588274 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.588203 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:29.588274 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.588219 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:29.588274 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.588234 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:29.588274 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.588248 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:29.589333 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.589309 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-envoy" pod "368378dd-a0ab-49f6-8c9d-8f457863b3ec" (UID: "368378dd-a0ab-49f6-8c9d-8f457863b3ec"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:29.589447 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.589423 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/368378dd-a0ab-49f6-8c9d-8f457863b3ec-kube-api-access-qqtzg" pod "368378dd-a0ab-49f6-8c9d-8f457863b3ec" (UID: "368378dd-a0ab-49f6-8c9d-8f457863b3ec"). InnerVolumeSpecName "kube-api-access-qqtzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:29.589561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.589479 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-podinfo" pod "368378dd-a0ab-49f6-8c9d-8f457863b3ec" (UID: "368378dd-a0ab-49f6-8c9d-8f457863b3ec"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:29.589643 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.589561 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-token" pod "368378dd-a0ab-49f6-8c9d-8f457863b3ec" (UID: "368378dd-a0ab-49f6-8c9d-8f457863b3ec"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:29.689614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.689537 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqtzg\" (UniqueName: \"kubernetes.io/projected/368378dd-a0ab-49f6-8c9d-8f457863b3ec-kube-api-access-qqtzg\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:29.689614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.689572 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:29.689614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.689586 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:29.689614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:29.689597 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/368378dd-a0ab-49f6-8c9d-8f457863b3ec-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:30.027819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.027781 5232 generic.go:356] "Generic (PLEG): container finished" podID="368378dd-a0ab-49f6-8c9d-8f457863b3ec" containerID="48b4a7fe8f0e88fd9fd2b05045a4e31c6b46164bff3895646f5261e88a6afb62" exitCode=0 Oct 30 18:27:30.028262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.027837 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" Oct 30 18:27:30.028262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.027862 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" event={"ID":"368378dd-a0ab-49f6-8c9d-8f457863b3ec","Type":"ContainerDied","Data":"48b4a7fe8f0e88fd9fd2b05045a4e31c6b46164bff3895646f5261e88a6afb62"} Oct 30 18:27:30.028262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.027916 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft" event={"ID":"368378dd-a0ab-49f6-8c9d-8f457863b3ec","Type":"ContainerDied","Data":"1d6ddc7a972e4067cd50c8e97ead030d9cb2307f7c746211086abaad231ba55e"} Oct 30 18:27:30.028262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.027942 5232 scope.go:122] "RemoveContainer" containerID="48b4a7fe8f0e88fd9fd2b05045a4e31c6b46164bff3895646f5261e88a6afb62" Oct 30 18:27:30.040379 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.040360 5232 scope.go:122] "RemoveContainer" containerID="2dbf7f69306cc194d28bf0b85ee1087a7edc9218d2e6de115a878477abb65bb7" Oct 30 18:27:30.051299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.051273 5232 scope.go:122] "RemoveContainer" containerID="48b4a7fe8f0e88fd9fd2b05045a4e31c6b46164bff3895646f5261e88a6afb62" Oct 30 18:27:30.051706 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:27:30.051685 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48b4a7fe8f0e88fd9fd2b05045a4e31c6b46164bff3895646f5261e88a6afb62\": container with ID starting with 48b4a7fe8f0e88fd9fd2b05045a4e31c6b46164bff3895646f5261e88a6afb62 not found: ID does not exist" containerID="48b4a7fe8f0e88fd9fd2b05045a4e31c6b46164bff3895646f5261e88a6afb62" Oct 30 18:27:30.051811 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.051710 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b4a7fe8f0e88fd9fd2b05045a4e31c6b46164bff3895646f5261e88a6afb62"} err="failed to get container status \"48b4a7fe8f0e88fd9fd2b05045a4e31c6b46164bff3895646f5261e88a6afb62\": rpc error: code = NotFound desc = could not find container \"48b4a7fe8f0e88fd9fd2b05045a4e31c6b46164bff3895646f5261e88a6afb62\": container with ID starting with 48b4a7fe8f0e88fd9fd2b05045a4e31c6b46164bff3895646f5261e88a6afb62 not found: ID does not exist" Oct 30 18:27:30.051811 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.051729 5232 scope.go:122] "RemoveContainer" containerID="2dbf7f69306cc194d28bf0b85ee1087a7edc9218d2e6de115a878477abb65bb7" Oct 30 18:27:30.051962 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:27:30.051943 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dbf7f69306cc194d28bf0b85ee1087a7edc9218d2e6de115a878477abb65bb7\": container with ID starting with 2dbf7f69306cc194d28bf0b85ee1087a7edc9218d2e6de115a878477abb65bb7 not found: ID does not exist" containerID="2dbf7f69306cc194d28bf0b85ee1087a7edc9218d2e6de115a878477abb65bb7" Oct 30 18:27:30.052021 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.051965 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dbf7f69306cc194d28bf0b85ee1087a7edc9218d2e6de115a878477abb65bb7"} err="failed to get container status \"2dbf7f69306cc194d28bf0b85ee1087a7edc9218d2e6de115a878477abb65bb7\": rpc error: code = NotFound desc = could not find container \"2dbf7f69306cc194d28bf0b85ee1087a7edc9218d2e6de115a878477abb65bb7\": container with ID starting with 2dbf7f69306cc194d28bf0b85ee1087a7edc9218d2e6de115a878477abb65bb7 not found: ID does not exist" Oct 30 18:27:30.082109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.082075 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft"] Oct 30 18:27:30.100558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.095975 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-fb0abbf7-openshift-default-cff7b9797-fjvft"] Oct 30 18:27:30.206627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.206602 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28" Oct 30 18:27:30.299575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.299456 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0963e410-ad61-4bae-b699-997384481b61-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0963e410-ad61-4bae-b699-997384481b61-maas-api-xks-ca\") pod \"0963e410-ad61-4bae-b699-997384481b61\" (UID: \"0963e410-ad61-4bae-b699-997384481b61\") " Oct 30 18:27:30.299729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.299659 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0963e410-ad61-4bae-b699-997384481b61-kube-api-access-fb846\" (UniqueName: \"kubernetes.io/projected/0963e410-ad61-4bae-b699-997384481b61-kube-api-access-fb846\") pod \"0963e410-ad61-4bae-b699-997384481b61\" (UID: \"0963e410-ad61-4bae-b699-997384481b61\") " Oct 30 18:27:30.299785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.299773 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0963e410-ad61-4bae-b699-997384481b61-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0963e410-ad61-4bae-b699-997384481b61-maas-api-openshift-ca\") pod \"0963e410-ad61-4bae-b699-997384481b61\" (UID: \"0963e410-ad61-4bae-b699-997384481b61\") " Oct 30 18:27:30.300144 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.300122 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0963e410-ad61-4bae-b699-997384481b61-maas-api-openshift-ca" pod "0963e410-ad61-4bae-b699-997384481b61" (UID: "0963e410-ad61-4bae-b699-997384481b61"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:30.300265 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.300250 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0963e410-ad61-4bae-b699-997384481b61-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:30.301697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.301676 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0963e410-ad61-4bae-b699-997384481b61-maas-api-xks-ca" pod "0963e410-ad61-4bae-b699-997384481b61" (UID: "0963e410-ad61-4bae-b699-997384481b61"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:30.301939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.301920 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0963e410-ad61-4bae-b699-997384481b61-kube-api-access-fb846" pod "0963e410-ad61-4bae-b699-997384481b61" (UID: "0963e410-ad61-4bae-b699-997384481b61"). InnerVolumeSpecName "kube-api-access-fb846". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:30.401884 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.401856 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fb846\" (UniqueName: \"kubernetes.io/projected/0963e410-ad61-4bae-b699-997384481b61-kube-api-access-fb846\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:30.401884 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.401881 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0963e410-ad61-4bae-b699-997384481b61-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:30.804268 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:30.804236 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="368378dd-a0ab-49f6-8c9d-8f457863b3ec" path="/var/lib/kubelet/pods/368378dd-a0ab-49f6-8c9d-8f457863b3ec/volumes" Oct 30 18:27:31.049129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.049097 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28" event={"ID":"0963e410-ad61-4bae-b699-997384481b61","Type":"ContainerDied","Data":"7116b2c8b8f3e7b728bb2f692c2b95f7833272079e5f9ab7dd9877a76844058c"} Oct 30 18:27:31.049129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.049126 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28" Oct 30 18:27:31.049584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.049167 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7116b2c8b8f3e7b728bb2f692c2b95f7833272079e5f9ab7dd9877a76844058c" Oct 30 18:27:31.350806 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.350745 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:27:31.456536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.443545 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-workload-socket\") pod \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " Oct 30 18:27:31.456536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.443658 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-kube-api-access-96dtv\" (UniqueName: \"kubernetes.io/projected/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-kube-api-access-96dtv\") pod \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " Oct 30 18:27:31.456536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.443760 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-credential-socket\") pod \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " Oct 30 18:27:31.456536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.443818 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-podinfo\") pod \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " Oct 30 18:27:31.456536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.443851 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istiod-ca-cert\") pod \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " Oct 30 18:27:31.456536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.443941 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-workload-certs\") pod \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " Oct 30 18:27:31.456536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.444028 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-envoy\") pod \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " Oct 30 18:27:31.456536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.444155 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-token\" (UniqueName: \"kubernetes.io/projected/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-token\") pod \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " Oct 30 18:27:31.456536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.444249 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-data\") pod \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\" (UID: \"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6\") " Oct 30 18:27:31.456536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.449303 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-data" pod "7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" (UID: "7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:31.457161 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.456637 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-workload-socket" pod "7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" (UID: "7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:31.464276 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.464210 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-workload-certs" pod "7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" (UID: "7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:31.464673 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.464421 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-credential-socket" pod "7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" (UID: "7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:31.464673 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.464636 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istiod-ca-cert" pod "7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" (UID: "7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:31.465740 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.465703 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-kube-api-access-96dtv" pod "7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" (UID: "7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6"). InnerVolumeSpecName "kube-api-access-96dtv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:31.467813 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.467783 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-podinfo" pod "7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" (UID: "7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:31.467985 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.467961 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-token" pod "7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" (UID: "7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:31.469586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.469495 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-envoy" pod "7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" (UID: "7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:31.551030 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.551001 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:31.551030 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.551028 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:31.551201 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.551038 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-96dtv\" (UniqueName: \"kubernetes.io/projected/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-kube-api-access-96dtv\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:31.551201 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.551048 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:31.551201 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.551056 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:31.551201 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.551064 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:31.551201 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.551073 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:31.551201 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.551080 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:31.551201 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:31.551087 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:32.057492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:32.057455 5232 generic.go:356] "Generic (PLEG): container finished" podID="7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" containerID="e298fbbad008039389123aaf40766a83939a57e825111f4254a3ae258f2b3cf2" exitCode=0 Oct 30 18:27:32.057957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:32.057503 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" Oct 30 18:27:32.057957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:32.057503 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" event={"ID":"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6","Type":"ContainerDied","Data":"e298fbbad008039389123aaf40766a83939a57e825111f4254a3ae258f2b3cf2"} Oct 30 18:27:32.057957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:32.057577 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2" event={"ID":"7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6","Type":"ContainerDied","Data":"244e624eb0e1d066097c550cff1f49f01e236b7aeebaeedf72faed216f5bce23"} Oct 30 18:27:32.057957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:32.057603 5232 scope.go:122] "RemoveContainer" containerID="e298fbbad008039389123aaf40766a83939a57e825111f4254a3ae258f2b3cf2" Oct 30 18:27:32.075653 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:32.075536 5232 scope.go:122] "RemoveContainer" containerID="760b4a2fa1a6e8045933313a9b9c2113acbddd616d424c879f758c55680d476e" Oct 30 18:27:32.085642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:32.085600 5232 scope.go:122] "RemoveContainer" containerID="e298fbbad008039389123aaf40766a83939a57e825111f4254a3ae258f2b3cf2" Oct 30 18:27:32.085918 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:27:32.085897 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e298fbbad008039389123aaf40766a83939a57e825111f4254a3ae258f2b3cf2\": container with ID starting with e298fbbad008039389123aaf40766a83939a57e825111f4254a3ae258f2b3cf2 not found: ID does not exist" containerID="e298fbbad008039389123aaf40766a83939a57e825111f4254a3ae258f2b3cf2" Oct 30 18:27:32.086015 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:32.085930 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e298fbbad008039389123aaf40766a83939a57e825111f4254a3ae258f2b3cf2"} err="failed to get container status \"e298fbbad008039389123aaf40766a83939a57e825111f4254a3ae258f2b3cf2\": rpc error: code = NotFound desc = could not find container \"e298fbbad008039389123aaf40766a83939a57e825111f4254a3ae258f2b3cf2\": container with ID starting with e298fbbad008039389123aaf40766a83939a57e825111f4254a3ae258f2b3cf2 not found: ID does not exist" Oct 30 18:27:32.086015 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:32.085953 5232 scope.go:122] "RemoveContainer" containerID="760b4a2fa1a6e8045933313a9b9c2113acbddd616d424c879f758c55680d476e" Oct 30 18:27:32.086249 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:27:32.086229 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"760b4a2fa1a6e8045933313a9b9c2113acbddd616d424c879f758c55680d476e\": container with ID starting with 760b4a2fa1a6e8045933313a9b9c2113acbddd616d424c879f758c55680d476e not found: ID does not exist" containerID="760b4a2fa1a6e8045933313a9b9c2113acbddd616d424c879f758c55680d476e" Oct 30 18:27:32.086297 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:32.086253 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"760b4a2fa1a6e8045933313a9b9c2113acbddd616d424c879f758c55680d476e"} err="failed to get container status \"760b4a2fa1a6e8045933313a9b9c2113acbddd616d424c879f758c55680d476e\": rpc error: code = NotFound desc = could not find container \"760b4a2fa1a6e8045933313a9b9c2113acbddd616d424c879f758c55680d476e\": container with ID starting with 760b4a2fa1a6e8045933313a9b9c2113acbddd616d424c879f758c55680d476e not found: ID does not exist" Oct 30 18:27:32.114769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:32.114747 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2"] Oct 30 18:27:32.126894 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:32.126875 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-9066aa-openshift-default-7b8fff5d7-5fsb2"] Oct 30 18:27:32.656338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:32.656302 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9"] Oct 30 18:27:32.656618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:32.656585 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" podUID="7dbbd730-9357-4f2f-b2d3-f41c877205e3" containerName="istio-proxy" containerID="cri-o://8875f64cfc317a51af45be8c0441d7bef0a6ad9e90277f9a219aeec6b1167ade" gracePeriod=30 Oct 30 18:27:32.841536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:32.841352 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" path="/var/lib/kubelet/pods/7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6/volumes" Oct 30 18:27:33.294815 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.294776 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk"] Oct 30 18:27:33.295496 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.295478 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" containerName="istio-proxy" Oct 30 18:27:33.295496 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.295495 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" containerName="istio-proxy" Oct 30 18:27:33.295725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.295557 5232 cpu_manager.go:414] "containerMap: removing container" podUID="368378dd-a0ab-49f6-8c9d-8f457863b3ec" containerName="istio-proxy" Oct 30 18:27:33.295725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.295565 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="368378dd-a0ab-49f6-8c9d-8f457863b3ec" containerName="istio-proxy" Oct 30 18:27:33.295725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.295581 5232 cpu_manager.go:414] "containerMap: removing container" podUID="368378dd-a0ab-49f6-8c9d-8f457863b3ec" containerName="istio-proxy" Oct 30 18:27:33.295725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.295587 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="368378dd-a0ab-49f6-8c9d-8f457863b3ec" containerName="istio-proxy" Oct 30 18:27:33.295725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.295601 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" containerName="istio-proxy" Oct 30 18:27:33.295725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.295605 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" containerName="istio-proxy" Oct 30 18:27:33.295725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.295621 5232 cpu_manager.go:414] "containerMap: removing container" podUID="0963e410-ad61-4bae-b699-997384481b61" containerName="revoke-keys" Oct 30 18:27:33.295725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.295625 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0963e410-ad61-4bae-b699-997384481b61" containerName="revoke-keys" Oct 30 18:27:33.296140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.295741 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" containerName="istio-proxy" Oct 30 18:27:33.296140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.295754 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="368378dd-a0ab-49f6-8c9d-8f457863b3ec" containerName="istio-proxy" Oct 30 18:27:33.296140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.295764 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="368378dd-a0ab-49f6-8c9d-8f457863b3ec" containerName="istio-proxy" Oct 30 18:27:33.296140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.295777 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dc14e97-fdc1-4a4c-9c76-f0f63cbb54b6" containerName="istio-proxy" Oct 30 18:27:33.296140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.295787 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="0963e410-ad61-4bae-b699-997384481b61" containerName="revoke-keys" Oct 30 18:27:33.301745 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.301724 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk" Oct 30 18:27:33.334544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.334492 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:27:33.334746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.334727 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:27:33.338942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.338911 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk"] Oct 30 18:27:33.516167 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.516136 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk\" (UID: \"af1b2984-d8ad-44a9-a5d9-85ea9efea63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk" Oct 30 18:27:33.516167 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.516177 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xb89v\" (UniqueName: \"kubernetes.io/projected/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-kube-api-access-xb89v\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk\" (UID: \"af1b2984-d8ad-44a9-a5d9-85ea9efea63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk" Oct 30 18:27:33.516378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.516340 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk\" (UID: \"af1b2984-d8ad-44a9-a5d9-85ea9efea63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk" Oct 30 18:27:33.618133 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.618051 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk\" (UID: \"af1b2984-d8ad-44a9-a5d9-85ea9efea63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk" Oct 30 18:27:33.618277 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.618257 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk\" (UID: \"af1b2984-d8ad-44a9-a5d9-85ea9efea63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk" Oct 30 18:27:33.618342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.618295 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xb89v\" (UniqueName: \"kubernetes.io/projected/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-kube-api-access-xb89v\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk\" (UID: \"af1b2984-d8ad-44a9-a5d9-85ea9efea63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk" Oct 30 18:27:33.618955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.618930 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk\" (UID: \"af1b2984-d8ad-44a9-a5d9-85ea9efea63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk" Oct 30 18:27:33.620659 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.620637 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk\" (UID: \"af1b2984-d8ad-44a9-a5d9-85ea9efea63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk" Oct 30 18:27:33.638581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.638559 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xb89v\" (UniqueName: \"kubernetes.io/projected/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-kube-api-access-xb89v\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk\" (UID: \"af1b2984-d8ad-44a9-a5d9-85ea9efea63a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk" Oct 30 18:27:33.643207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.643189 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk" Oct 30 18:27:33.943888 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:27:33.943851 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaf1b2984_d8ad_44a9_a5d9_85ea9efea63a.slice/crio-7635fcc10b3ba47529ae0e092d934dd0a03a2da25edef93a4deb6d66d6ca46b1 WatchSource:0}: Error finding container 7635fcc10b3ba47529ae0e092d934dd0a03a2da25edef93a4deb6d66d6ca46b1: Status 404 returned error can't find the container with id 7635fcc10b3ba47529ae0e092d934dd0a03a2da25edef93a4deb6d66d6ca46b1 Oct 30 18:27:33.944817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:33.944796 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk"] Oct 30 18:27:34.076799 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:34.076762 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk" event={"ID":"af1b2984-d8ad-44a9-a5d9-85ea9efea63a","Type":"ContainerStarted","Data":"7635fcc10b3ba47529ae0e092d934dd0a03a2da25edef93a4deb6d66d6ca46b1"} Oct 30 18:27:35.086811 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:35.086728 5232 generic.go:356] "Generic (PLEG): container finished" podID="af1b2984-d8ad-44a9-a5d9-85ea9efea63a" containerID="6f2ea9369e7c1c2866201e31b1bdaf103d68014babd6f8446bc1f9eb85df91dc" exitCode=0 Oct 30 18:27:35.087207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:35.086820 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk" event={"ID":"af1b2984-d8ad-44a9-a5d9-85ea9efea63a","Type":"ContainerDied","Data":"6f2ea9369e7c1c2866201e31b1bdaf103d68014babd6f8446bc1f9eb85df91dc"} Oct 30 18:27:36.270499 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:36.270471 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk" Oct 30 18:27:36.360536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:36.360431 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-maas-api-openshift-ca\") pod \"af1b2984-d8ad-44a9-a5d9-85ea9efea63a\" (UID: \"af1b2984-d8ad-44a9-a5d9-85ea9efea63a\") " Oct 30 18:27:36.360536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:36.360492 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-maas-api-xks-ca\") pod \"af1b2984-d8ad-44a9-a5d9-85ea9efea63a\" (UID: \"af1b2984-d8ad-44a9-a5d9-85ea9efea63a\") " Oct 30 18:27:36.360792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:36.360579 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-kube-api-access-xb89v\" (UniqueName: \"kubernetes.io/projected/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-kube-api-access-xb89v\") pod \"af1b2984-d8ad-44a9-a5d9-85ea9efea63a\" (UID: \"af1b2984-d8ad-44a9-a5d9-85ea9efea63a\") " Oct 30 18:27:36.360901 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:36.360873 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-maas-api-openshift-ca" pod "af1b2984-d8ad-44a9-a5d9-85ea9efea63a" (UID: "af1b2984-d8ad-44a9-a5d9-85ea9efea63a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:36.361346 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:36.361279 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:36.363022 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:36.362983 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-maas-api-xks-ca" pod "af1b2984-d8ad-44a9-a5d9-85ea9efea63a" (UID: "af1b2984-d8ad-44a9-a5d9-85ea9efea63a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:36.363131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:36.363098 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-kube-api-access-xb89v" pod "af1b2984-d8ad-44a9-a5d9-85ea9efea63a" (UID: "af1b2984-d8ad-44a9-a5d9-85ea9efea63a"). InnerVolumeSpecName "kube-api-access-xb89v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:36.462845 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:36.462815 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:36.462845 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:36.462844 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xb89v\" (UniqueName: \"kubernetes.io/projected/af1b2984-d8ad-44a9-a5d9-85ea9efea63a-kube-api-access-xb89v\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:37.102060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:37.101983 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk" Oct 30 18:27:37.102060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:37.102014 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk" event={"ID":"af1b2984-d8ad-44a9-a5d9-85ea9efea63a","Type":"ContainerDied","Data":"7635fcc10b3ba47529ae0e092d934dd0a03a2da25edef93a4deb6d66d6ca46b1"} Oct 30 18:27:37.102060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:37.102062 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7635fcc10b3ba47529ae0e092d934dd0a03a2da25edef93a4deb6d66d6ca46b1" Oct 30 18:27:38.113000 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.112967 5232 generic.go:356] "Generic (PLEG): container finished" podID="7dbbd730-9357-4f2f-b2d3-f41c877205e3" containerID="8875f64cfc317a51af45be8c0441d7bef0a6ad9e90277f9a219aeec6b1167ade" exitCode=0 Oct 30 18:27:38.113503 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.113040 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" event={"ID":"7dbbd730-9357-4f2f-b2d3-f41c877205e3","Type":"ContainerDied","Data":"8875f64cfc317a51af45be8c0441d7bef0a6ad9e90277f9a219aeec6b1167ade"} Oct 30 18:27:38.113503 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.113118 5232 scope.go:122] "RemoveContainer" containerID="ef915c07cd6bb50f797e5174e691fd7981623d37390a50fc7a2934a335142f26" Oct 30 18:27:38.354381 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.354356 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:27:38.397680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.397614 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-envoy\") pod \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " Oct 30 18:27:38.397819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.397714 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-podinfo\") pod \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " Oct 30 18:27:38.397819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.397795 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-workload-socket\") pod \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " Oct 30 18:27:38.397945 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.397837 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-credential-socket\") pod \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " Oct 30 18:27:38.397945 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.397902 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-token\" (UniqueName: \"kubernetes.io/projected/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-token\") pod \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " Oct 30 18:27:38.398065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.397944 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-workload-certs\") pod \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " Oct 30 18:27:38.398121 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.398086 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7dbbd730-9357-4f2f-b2d3-f41c877205e3-kube-api-access-6xczq\" (UniqueName: \"kubernetes.io/projected/7dbbd730-9357-4f2f-b2d3-f41c877205e3-kube-api-access-6xczq\") pod \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " Oct 30 18:27:38.398188 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.398115 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-credential-socket" pod "7dbbd730-9357-4f2f-b2d3-f41c877205e3" (UID: "7dbbd730-9357-4f2f-b2d3-f41c877205e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:38.398188 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.398121 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-workload-socket" pod "7dbbd730-9357-4f2f-b2d3-f41c877205e3" (UID: "7dbbd730-9357-4f2f-b2d3-f41c877205e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:38.398188 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.398173 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istiod-ca-cert\") pod \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " Oct 30 18:27:38.398350 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.398227 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-data\") pod \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\" (UID: \"7dbbd730-9357-4f2f-b2d3-f41c877205e3\") " Oct 30 18:27:38.398350 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.398340 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-workload-certs" pod "7dbbd730-9357-4f2f-b2d3-f41c877205e3" (UID: "7dbbd730-9357-4f2f-b2d3-f41c877205e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:38.398808 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.398786 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-data" pod "7dbbd730-9357-4f2f-b2d3-f41c877205e3" (UID: "7dbbd730-9357-4f2f-b2d3-f41c877205e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:38.398963 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.398941 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:38.399039 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.398968 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:38.399039 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.398982 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:38.399039 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.398995 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:38.399206 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.399147 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istiod-ca-cert" pod "7dbbd730-9357-4f2f-b2d3-f41c877205e3" (UID: "7dbbd730-9357-4f2f-b2d3-f41c877205e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:38.400220 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.400194 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-podinfo" pod "7dbbd730-9357-4f2f-b2d3-f41c877205e3" (UID: "7dbbd730-9357-4f2f-b2d3-f41c877205e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:38.400439 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.400401 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-envoy" pod "7dbbd730-9357-4f2f-b2d3-f41c877205e3" (UID: "7dbbd730-9357-4f2f-b2d3-f41c877205e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:38.400775 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.400755 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dbbd730-9357-4f2f-b2d3-f41c877205e3-kube-api-access-6xczq" pod "7dbbd730-9357-4f2f-b2d3-f41c877205e3" (UID: "7dbbd730-9357-4f2f-b2d3-f41c877205e3"). InnerVolumeSpecName "kube-api-access-6xczq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:38.401005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.400987 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-token" pod "7dbbd730-9357-4f2f-b2d3-f41c877205e3" (UID: "7dbbd730-9357-4f2f-b2d3-f41c877205e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:38.512534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.507907 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:38.512534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.507948 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:38.512534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.507963 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6xczq\" (UniqueName: \"kubernetes.io/projected/7dbbd730-9357-4f2f-b2d3-f41c877205e3-kube-api-access-6xczq\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:38.512534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.507978 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:38.512534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:38.507992 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7dbbd730-9357-4f2f-b2d3-f41c877205e3-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:39.122157 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:39.122074 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" Oct 30 18:27:39.122157 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:39.122132 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9" event={"ID":"7dbbd730-9357-4f2f-b2d3-f41c877205e3","Type":"ContainerDied","Data":"506baaa497bea0e354f427c099515a0bd053fa4b230eb67b71e200fc6ca97f9c"} Oct 30 18:27:39.122749 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:39.122188 5232 scope.go:122] "RemoveContainer" containerID="8875f64cfc317a51af45be8c0441d7bef0a6ad9e90277f9a219aeec6b1167ade" Oct 30 18:27:39.187014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:39.186982 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9"] Oct 30 18:27:39.198978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:39.198948 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-18cd5a-openshift-default-794dd87655-c4km9"] Oct 30 18:27:40.804574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:40.804544 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7dbbd730-9357-4f2f-b2d3-f41c877205e3" path="/var/lib/kubelet/pods/7dbbd730-9357-4f2f-b2d3-f41c877205e3/volumes" Oct 30 18:27:43.520704 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.520668 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn"] Oct 30 18:27:43.521339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.521323 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7dbbd730-9357-4f2f-b2d3-f41c877205e3" containerName="istio-proxy" Oct 30 18:27:43.521384 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.521340 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dbbd730-9357-4f2f-b2d3-f41c877205e3" containerName="istio-proxy" Oct 30 18:27:43.521427 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.521384 5232 cpu_manager.go:414] "containerMap: removing container" podUID="af1b2984-d8ad-44a9-a5d9-85ea9efea63a" containerName="revoke-keys" Oct 30 18:27:43.521427 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.521389 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af1b2984-d8ad-44a9-a5d9-85ea9efea63a" containerName="revoke-keys" Oct 30 18:27:43.521492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.521479 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dbbd730-9357-4f2f-b2d3-f41c877205e3" containerName="istio-proxy" Oct 30 18:27:43.521492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.521491 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="af1b2984-d8ad-44a9-a5d9-85ea9efea63a" containerName="revoke-keys" Oct 30 18:27:43.521645 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.521500 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7dbbd730-9357-4f2f-b2d3-f41c877205e3" containerName="istio-proxy" Oct 30 18:27:43.526651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.526629 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn" Oct 30 18:27:43.543333 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.543304 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn"] Oct 30 18:27:43.550697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.550672 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:27:43.550870 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.550785 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:27:43.685293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.685256 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb32688-9676-4e30-8b67-61d6344b4d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aa66234-f85dn\" (UID: \"6cb32688-9676-4e30-8b67-61d6344b4d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn" Oct 30 18:27:43.685474 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.685376 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb32688-9676-4e30-8b67-61d6344b4d8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aa66234-f85dn\" (UID: \"6cb32688-9676-4e30-8b67-61d6344b4d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn" Oct 30 18:27:43.685617 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.685594 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qg8q7\" (UniqueName: \"kubernetes.io/projected/6cb32688-9676-4e30-8b67-61d6344b4d8d-kube-api-access-qg8q7\") pod \"maas-api-revoke-sub-3aa66234-f85dn\" (UID: \"6cb32688-9676-4e30-8b67-61d6344b4d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn" Oct 30 18:27:43.716215 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.716183 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn"] Oct 30 18:27:43.779471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.779388 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg"] Oct 30 18:27:43.787368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.787339 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qg8q7\" (UniqueName: \"kubernetes.io/projected/6cb32688-9676-4e30-8b67-61d6344b4d8d-kube-api-access-qg8q7\") pod \"maas-api-revoke-sub-3aa66234-f85dn\" (UID: \"6cb32688-9676-4e30-8b67-61d6344b4d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn" Oct 30 18:27:43.787519 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.787406 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb32688-9676-4e30-8b67-61d6344b4d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aa66234-f85dn\" (UID: \"6cb32688-9676-4e30-8b67-61d6344b4d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn" Oct 30 18:27:43.787577 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.787502 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb32688-9676-4e30-8b67-61d6344b4d8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aa66234-f85dn\" (UID: \"6cb32688-9676-4e30-8b67-61d6344b4d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn" Oct 30 18:27:43.788108 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.788089 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb32688-9676-4e30-8b67-61d6344b4d8d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3aa66234-f85dn\" (UID: \"6cb32688-9676-4e30-8b67-61d6344b4d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn" Oct 30 18:27:43.789979 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.789962 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb32688-9676-4e30-8b67-61d6344b4d8d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3aa66234-f85dn\" (UID: \"6cb32688-9676-4e30-8b67-61d6344b4d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn" Oct 30 18:27:43.807072 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.807036 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qg8q7\" (UniqueName: \"kubernetes.io/projected/6cb32688-9676-4e30-8b67-61d6344b4d8d-kube-api-access-qg8q7\") pod \"maas-api-revoke-sub-3aa66234-f85dn\" (UID: \"6cb32688-9676-4e30-8b67-61d6344b4d8d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn" Oct 30 18:27:43.842481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:43.842445 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn" Oct 30 18:27:44.158481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:44.158447 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn"] Oct 30 18:27:44.158739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:27:44.158715 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cb32688_9676_4e30_8b67_61d6344b4d8d.slice/crio-541bc839f6fddcdd970f66180440fc1e535d9971018d5cee5a0a855b257599ed WatchSource:0}: Error finding container 541bc839f6fddcdd970f66180440fc1e535d9971018d5cee5a0a855b257599ed: Status 404 returned error can't find the container with id 541bc839f6fddcdd970f66180440fc1e535d9971018d5cee5a0a855b257599ed Oct 30 18:27:45.203702 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:45.202742 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn" event={"ID":"6cb32688-9676-4e30-8b67-61d6344b4d8d","Type":"ContainerStarted","Data":"ef21f62165cf1bd5133d84a4fa604bb51cb7c64e9b7cb45ae376d649dd3e402d"} Oct 30 18:27:45.203702 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:45.202812 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn" event={"ID":"6cb32688-9676-4e30-8b67-61d6344b4d8d","Type":"ContainerStarted","Data":"541bc839f6fddcdd970f66180440fc1e535d9971018d5cee5a0a855b257599ed"} Oct 30 18:27:45.262653 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:45.248431 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn" podStartSLOduration=2.24840939 podStartE2EDuration="2.24840939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:27:45.247860239 +0000 UTC m=+1564.564280563" watchObservedRunningTime="2026-10-30 18:27:45.24840939 +0000 UTC m=+1564.564829716" Oct 30 18:27:46.260804 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:46.260761 5232 generic.go:356] "Generic (PLEG): container finished" podID="6cb32688-9676-4e30-8b67-61d6344b4d8d" containerID="ef21f62165cf1bd5133d84a4fa604bb51cb7c64e9b7cb45ae376d649dd3e402d" exitCode=0 Oct 30 18:27:46.261215 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:46.260842 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn" event={"ID":"6cb32688-9676-4e30-8b67-61d6344b4d8d","Type":"ContainerDied","Data":"ef21f62165cf1bd5133d84a4fa604bb51cb7c64e9b7cb45ae376d649dd3e402d"} Oct 30 18:27:47.550084 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:47.518609 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn" Oct 30 18:27:47.603440 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:47.603403 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cb32688-9676-4e30-8b67-61d6344b4d8d-kube-api-access-qg8q7\" (UniqueName: \"kubernetes.io/projected/6cb32688-9676-4e30-8b67-61d6344b4d8d-kube-api-access-qg8q7\") pod \"6cb32688-9676-4e30-8b67-61d6344b4d8d\" (UID: \"6cb32688-9676-4e30-8b67-61d6344b4d8d\") " Oct 30 18:27:47.603738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:47.603716 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cb32688-9676-4e30-8b67-61d6344b4d8d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb32688-9676-4e30-8b67-61d6344b4d8d-maas-api-openshift-ca\") pod \"6cb32688-9676-4e30-8b67-61d6344b4d8d\" (UID: \"6cb32688-9676-4e30-8b67-61d6344b4d8d\") " Oct 30 18:27:47.604037 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:47.604018 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cb32688-9676-4e30-8b67-61d6344b4d8d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb32688-9676-4e30-8b67-61d6344b4d8d-maas-api-xks-ca\") pod \"6cb32688-9676-4e30-8b67-61d6344b4d8d\" (UID: \"6cb32688-9676-4e30-8b67-61d6344b4d8d\") " Oct 30 18:27:47.604424 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:47.604395 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cb32688-9676-4e30-8b67-61d6344b4d8d-maas-api-openshift-ca" pod "6cb32688-9676-4e30-8b67-61d6344b4d8d" (UID: "6cb32688-9676-4e30-8b67-61d6344b4d8d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:47.604967 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:47.604707 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cb32688-9676-4e30-8b67-61d6344b4d8d-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:47.606214 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:47.606188 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cb32688-9676-4e30-8b67-61d6344b4d8d-kube-api-access-qg8q7" pod "6cb32688-9676-4e30-8b67-61d6344b4d8d" (UID: "6cb32688-9676-4e30-8b67-61d6344b4d8d"). InnerVolumeSpecName "kube-api-access-qg8q7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:47.607500 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:47.607476 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cb32688-9676-4e30-8b67-61d6344b4d8d-maas-api-xks-ca" pod "6cb32688-9676-4e30-8b67-61d6344b4d8d" (UID: "6cb32688-9676-4e30-8b67-61d6344b4d8d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:27:47.706302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:47.706268 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qg8q7\" (UniqueName: \"kubernetes.io/projected/6cb32688-9676-4e30-8b67-61d6344b4d8d-kube-api-access-qg8q7\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:47.706302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:47.706296 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cb32688-9676-4e30-8b67-61d6344b4d8d-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:48.278244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:48.278214 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn" Oct 30 18:27:48.278428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:48.278249 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn" event={"ID":"6cb32688-9676-4e30-8b67-61d6344b4d8d","Type":"ContainerDied","Data":"541bc839f6fddcdd970f66180440fc1e535d9971018d5cee5a0a855b257599ed"} Oct 30 18:27:48.278428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:48.278293 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="541bc839f6fddcdd970f66180440fc1e535d9971018d5cee5a0a855b257599ed" Oct 30 18:27:48.717832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:48.717573 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn" podUID="809fec18-40ef-42e0-9914-ff3b301068c0" containerName="payload-processing" containerID="cri-o://edfd1a9eb5dcf5ea9a117d07a420ce58f04ac17d9d1fe785c79bba184ae86b26" gracePeriod=25 Oct 30 18:27:48.780204 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:48.780170 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg" podUID="fc0db16f-42fd-4d0f-8549-33c9d7511b3d" containerName="payload-pre-processing" containerID="cri-o://37bde7ea3551597aa174283c8d326f012a852fa68c5d123407ac3ca861e722a8" gracePeriod=25 Oct 30 18:27:49.292765 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:49.292571 5232 generic.go:356] "Generic (PLEG): container finished" podID="fc0db16f-42fd-4d0f-8549-33c9d7511b3d" containerID="37bde7ea3551597aa174283c8d326f012a852fa68c5d123407ac3ca861e722a8" exitCode=0 Oct 30 18:27:49.292765 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:49.292699 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg" event={"ID":"fc0db16f-42fd-4d0f-8549-33c9d7511b3d","Type":"ContainerDied","Data":"37bde7ea3551597aa174283c8d326f012a852fa68c5d123407ac3ca861e722a8"} Oct 30 18:27:49.294648 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:49.294524 5232 generic.go:356] "Generic (PLEG): container finished" podID="809fec18-40ef-42e0-9914-ff3b301068c0" containerID="edfd1a9eb5dcf5ea9a117d07a420ce58f04ac17d9d1fe785c79bba184ae86b26" exitCode=0 Oct 30 18:27:49.294648 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:49.294602 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn" event={"ID":"809fec18-40ef-42e0-9914-ff3b301068c0","Type":"ContainerDied","Data":"edfd1a9eb5dcf5ea9a117d07a420ce58f04ac17d9d1fe785c79bba184ae86b26"} Oct 30 18:27:49.466382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:49.466356 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn" Oct 30 18:27:49.468501 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:49.468477 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg" Oct 30 18:27:49.575531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:49.575406 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc0db16f-42fd-4d0f-8549-33c9d7511b3d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0db16f-42fd-4d0f-8549-33c9d7511b3d-plugins-config-volume\") pod \"fc0db16f-42fd-4d0f-8549-33c9d7511b3d\" (UID: \"fc0db16f-42fd-4d0f-8549-33c9d7511b3d\") " Oct 30 18:27:49.575790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:49.575539 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc0db16f-42fd-4d0f-8549-33c9d7511b3d-kube-api-access-6t8fm\" (UniqueName: \"kubernetes.io/projected/fc0db16f-42fd-4d0f-8549-33c9d7511b3d-kube-api-access-6t8fm\") pod \"fc0db16f-42fd-4d0f-8549-33c9d7511b3d\" (UID: \"fc0db16f-42fd-4d0f-8549-33c9d7511b3d\") " Oct 30 18:27:49.575790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:49.575700 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/809fec18-40ef-42e0-9914-ff3b301068c0-kube-api-access-9htlc\" (UniqueName: \"kubernetes.io/projected/809fec18-40ef-42e0-9914-ff3b301068c0-kube-api-access-9htlc\") pod \"809fec18-40ef-42e0-9914-ff3b301068c0\" (UID: \"809fec18-40ef-42e0-9914-ff3b301068c0\") " Oct 30 18:27:49.575958 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:49.575933 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/809fec18-40ef-42e0-9914-ff3b301068c0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809fec18-40ef-42e0-9914-ff3b301068c0-plugins-config-volume\") pod \"809fec18-40ef-42e0-9914-ff3b301068c0\" (UID: \"809fec18-40ef-42e0-9914-ff3b301068c0\") " Oct 30 18:27:49.576956 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:49.576925 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/809fec18-40ef-42e0-9914-ff3b301068c0-plugins-config-volume" pod "809fec18-40ef-42e0-9914-ff3b301068c0" (UID: "809fec18-40ef-42e0-9914-ff3b301068c0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:49.577463 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:49.577441 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc0db16f-42fd-4d0f-8549-33c9d7511b3d-plugins-config-volume" pod "fc0db16f-42fd-4d0f-8549-33c9d7511b3d" (UID: "fc0db16f-42fd-4d0f-8549-33c9d7511b3d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:49.579928 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:49.579903 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/809fec18-40ef-42e0-9914-ff3b301068c0-kube-api-access-9htlc" pod "809fec18-40ef-42e0-9914-ff3b301068c0" (UID: "809fec18-40ef-42e0-9914-ff3b301068c0"). InnerVolumeSpecName "kube-api-access-9htlc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:49.581675 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:49.581647 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc0db16f-42fd-4d0f-8549-33c9d7511b3d-kube-api-access-6t8fm" pod "fc0db16f-42fd-4d0f-8549-33c9d7511b3d" (UID: "fc0db16f-42fd-4d0f-8549-33c9d7511b3d"). InnerVolumeSpecName "kube-api-access-6t8fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:49.678190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:49.678157 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/809fec18-40ef-42e0-9914-ff3b301068c0-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:49.678190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:49.678188 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc0db16f-42fd-4d0f-8549-33c9d7511b3d-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:49.678190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:49.678197 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6t8fm\" (UniqueName: \"kubernetes.io/projected/fc0db16f-42fd-4d0f-8549-33c9d7511b3d-kube-api-access-6t8fm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:49.678494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:49.678206 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9htlc\" (UniqueName: \"kubernetes.io/projected/809fec18-40ef-42e0-9914-ff3b301068c0-kube-api-access-9htlc\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:50.305488 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:50.305447 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn" event={"ID":"809fec18-40ef-42e0-9914-ff3b301068c0","Type":"ContainerDied","Data":"130dc2afb0e95a94214a22078aede33d56bfe8979c0e5b91cdeeb30a73ac8f66"} Oct 30 18:27:50.306045 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:50.305536 5232 scope.go:122] "RemoveContainer" containerID="edfd1a9eb5dcf5ea9a117d07a420ce58f04ac17d9d1fe785c79bba184ae86b26" Oct 30 18:27:50.306045 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:50.305653 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn" Oct 30 18:27:50.405053 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:50.405010 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg" event={"ID":"fc0db16f-42fd-4d0f-8549-33c9d7511b3d","Type":"ContainerDied","Data":"1e7a4a3de7bb80b738cf8f3c458814d32953c0805a0ed37fee1b7385a84d23f0"} Oct 30 18:27:50.428813 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:50.405607 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg" Oct 30 18:27:50.652682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:50.652650 5232 scope.go:122] "RemoveContainer" containerID="37bde7ea3551597aa174283c8d326f012a852fa68c5d123407ac3ca861e722a8" Oct 30 18:27:50.654315 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:50.654285 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn"] Oct 30 18:27:50.659995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:50.659966 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w0-ae3373-656cd95b64-vk6wn"] Oct 30 18:27:50.699637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:50.699608 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg"] Oct 30 18:27:50.703068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:50.703033 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w0-ae3373-d7558c9d7-gflgg"] Oct 30 18:27:50.805736 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:50.805701 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="809fec18-40ef-42e0-9914-ff3b301068c0" path="/var/lib/kubelet/pods/809fec18-40ef-42e0-9914-ff3b301068c0/volumes" Oct 30 18:27:50.806338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:50.806319 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc0db16f-42fd-4d0f-8549-33c9d7511b3d" path="/var/lib/kubelet/pods/fc0db16f-42fd-4d0f-8549-33c9d7511b3d/volumes" Oct 30 18:27:51.720209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:51.720171 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z"] Oct 30 18:27:51.720730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:51.720428 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" podUID="24aaf415-4614-4e59-87db-71cac71da3d9" containerName="istio-proxy" containerID="cri-o://9c51eb8456d05e5e6c2a23831daf3279f69d249bad8c82a3aac8f93a0171a1b6" gracePeriod=30 Oct 30 18:27:53.210532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:53.206662 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt"] Oct 30 18:27:53.220534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:53.216875 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-550276-k26kt"] Oct 30 18:27:54.337728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:54.337684 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:27:54.338176 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:54.337760 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" podUID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:54.357722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:54.357689 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:27:54.357854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:54.357736 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" podUID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:54.822974 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:54.822937 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ac3d81c-13cd-4273-b143-7b78579b24cb" path="/var/lib/kubelet/pods/2ac3d81c-13cd-4273-b143-7b78579b24cb/volumes" Oct 30 18:27:56.501781 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:56.498364 5232 generic.go:356] "Generic (PLEG): container finished" podID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerID="0e97b95f73eecec35274b0726fc5500f57a6051ebfa5dc281f87edf566737e47" exitCode=0 Oct 30 18:27:56.501781 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:56.498637 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" event={"ID":"37acc5cd-e689-4f1f-bb72-38794eb5e205","Type":"ContainerDied","Data":"0e97b95f73eecec35274b0726fc5500f57a6051ebfa5dc281f87edf566737e47"} Oct 30 18:27:56.501781 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:56.499255 5232 scope.go:122] "RemoveContainer" containerID="0e97b95f73eecec35274b0726fc5500f57a6051ebfa5dc281f87edf566737e47" Oct 30 18:27:56.511850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:56.510106 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:27:56.511850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:56.510222 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:27:56.511850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:56.510274 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:27:56.519661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:56.518213 5232 generic.go:356] "Generic (PLEG): container finished" podID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerID="c20ab9c1c9576eb5e86497bb8ec8e3ccd1787a00cfb6798231e7d1bb123f8455" exitCode=0 Oct 30 18:27:56.519661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:56.518438 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" event={"ID":"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998","Type":"ContainerDied","Data":"c20ab9c1c9576eb5e86497bb8ec8e3ccd1787a00cfb6798231e7d1bb123f8455"} Oct 30 18:27:56.519661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:56.519035 5232 scope.go:122] "RemoveContainer" containerID="c20ab9c1c9576eb5e86497bb8ec8e3ccd1787a00cfb6798231e7d1bb123f8455" Oct 30 18:27:56.525561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:56.522426 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:27:56.527048 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:56.525960 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:27:56.527048 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:56.526042 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:27:57.135552 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:27:57.125720 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:27:58.131544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.107411 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:27:58.131544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.108395 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" event={"ID":"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998","Type":"ContainerStarted","Data":"f2767476296f1939812236ce78d03b4f0677055fa6bdde7665176edac0a7a606"} Oct 30 18:27:58.177583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.162238 5232 generic.go:356] "Generic (PLEG): container finished" podID="24aaf415-4614-4e59-87db-71cac71da3d9" containerID="9c51eb8456d05e5e6c2a23831daf3279f69d249bad8c82a3aac8f93a0171a1b6" exitCode=0 Oct 30 18:27:58.177583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.162561 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" event={"ID":"24aaf415-4614-4e59-87db-71cac71da3d9","Type":"ContainerDied","Data":"9c51eb8456d05e5e6c2a23831daf3279f69d249bad8c82a3aac8f93a0171a1b6"} Oct 30 18:27:58.177583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.162641 5232 scope.go:122] "RemoveContainer" containerID="9c51eb8456d05e5e6c2a23831daf3279f69d249bad8c82a3aac8f93a0171a1b6" Oct 30 18:27:58.177583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.162870 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" Oct 30 18:27:58.228537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.228235 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" event={"ID":"37acc5cd-e689-4f1f-bb72-38794eb5e205","Type":"ContainerStarted","Data":"fbe4c68188085bdea85053a1d780b92618cad40a8eccaf2bc2fda31b377c5341"} Oct 30 18:27:58.257240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.257201 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-istio-envoy\") pod \"24aaf415-4614-4e59-87db-71cac71da3d9\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " Oct 30 18:27:58.257644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.257620 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-workload-certs\") pod \"24aaf415-4614-4e59-87db-71cac71da3d9\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " Oct 30 18:27:58.257862 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.257844 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24aaf415-4614-4e59-87db-71cac71da3d9-istio-token\" (UniqueName: \"kubernetes.io/projected/24aaf415-4614-4e59-87db-71cac71da3d9-istio-token\") pod \"24aaf415-4614-4e59-87db-71cac71da3d9\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " Oct 30 18:27:58.257988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.257973 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-istio-data\") pod \"24aaf415-4614-4e59-87db-71cac71da3d9\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " Oct 30 18:27:58.258120 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.258104 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-workload-socket\") pod \"24aaf415-4614-4e59-87db-71cac71da3d9\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " Oct 30 18:27:58.258264 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.258249 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24aaf415-4614-4e59-87db-71cac71da3d9-kube-api-access-rrr6d\" (UniqueName: \"kubernetes.io/projected/24aaf415-4614-4e59-87db-71cac71da3d9-kube-api-access-rrr6d\") pod \"24aaf415-4614-4e59-87db-71cac71da3d9\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " Oct 30 18:27:58.258413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.258398 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-credential-socket\") pod \"24aaf415-4614-4e59-87db-71cac71da3d9\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " Oct 30 18:27:58.258614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.258597 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24aaf415-4614-4e59-87db-71cac71da3d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24aaf415-4614-4e59-87db-71cac71da3d9-istio-podinfo\") pod \"24aaf415-4614-4e59-87db-71cac71da3d9\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " Oct 30 18:27:58.258817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.258800 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24aaf415-4614-4e59-87db-71cac71da3d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24aaf415-4614-4e59-87db-71cac71da3d9-istiod-ca-cert\") pod \"24aaf415-4614-4e59-87db-71cac71da3d9\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " Oct 30 18:27:58.266342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.266315 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-credential-socket" pod "24aaf415-4614-4e59-87db-71cac71da3d9" (UID: "24aaf415-4614-4e59-87db-71cac71da3d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:58.286658 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.286615 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-workload-certs" pod "24aaf415-4614-4e59-87db-71cac71da3d9" (UID: "24aaf415-4614-4e59-87db-71cac71da3d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:58.351787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.351744 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24aaf415-4614-4e59-87db-71cac71da3d9-istiod-ca-cert" pod "24aaf415-4614-4e59-87db-71cac71da3d9" (UID: "24aaf415-4614-4e59-87db-71cac71da3d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:27:58.352709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.352684 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-workload-socket" pod "24aaf415-4614-4e59-87db-71cac71da3d9" (UID: "24aaf415-4614-4e59-87db-71cac71da3d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:58.353057 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.353038 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-istio-data" pod "24aaf415-4614-4e59-87db-71cac71da3d9" (UID: "24aaf415-4614-4e59-87db-71cac71da3d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:58.388586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.382659 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/24aaf415-4614-4e59-87db-71cac71da3d9-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:58.388586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.382697 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:58.388586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.382711 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:58.388586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.382726 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:58.388586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.382741 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:58.722591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.715095 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:58.740690 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.739621 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:58.818546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.817619 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:58.841538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.838284 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24aaf415-4614-4e59-87db-71cac71da3d9-kube-api-access-rrr6d" pod "24aaf415-4614-4e59-87db-71cac71da3d9" (UID: "24aaf415-4614-4e59-87db-71cac71da3d9"). InnerVolumeSpecName "kube-api-access-rrr6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:58.867557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.861251 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24aaf415-4614-4e59-87db-71cac71da3d9-istio-podinfo" pod "24aaf415-4614-4e59-87db-71cac71da3d9" (UID: "24aaf415-4614-4e59-87db-71cac71da3d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:58.918618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.872143 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-istio-envoy" pod "24aaf415-4614-4e59-87db-71cac71da3d9" (UID: "24aaf415-4614-4e59-87db-71cac71da3d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:27:58.918618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.886452 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24aaf415-4614-4e59-87db-71cac71da3d9-istio-token" pod "24aaf415-4614-4e59-87db-71cac71da3d9" (UID: "24aaf415-4614-4e59-87db-71cac71da3d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:58.918618 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:27:58.901644 5232 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/24aaf415-4614-4e59-87db-71cac71da3d9/volumes/kubernetes.io~projected/kube-api-access-rrr6d Oct 30 18:27:58.918618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.911654 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24aaf415-4614-4e59-87db-71cac71da3d9-kube-api-access-rrr6d" pod "24aaf415-4614-4e59-87db-71cac71da3d9" (UID: "24aaf415-4614-4e59-87db-71cac71da3d9"). InnerVolumeSpecName "kube-api-access-rrr6d". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:27:58.918618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.886289 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:58.918618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.901447 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24aaf415-4614-4e59-87db-71cac71da3d9-kube-api-access-rrr6d\" (UniqueName: \"kubernetes.io/projected/24aaf415-4614-4e59-87db-71cac71da3d9-kube-api-access-rrr6d\") pod \"24aaf415-4614-4e59-87db-71cac71da3d9\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " Oct 30 18:27:58.918618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.912185 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/24aaf415-4614-4e59-87db-71cac71da3d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24aaf415-4614-4e59-87db-71cac71da3d9-istio-podinfo\") pod \"24aaf415-4614-4e59-87db-71cac71da3d9\" (UID: \"24aaf415-4614-4e59-87db-71cac71da3d9\") " Oct 30 18:27:58.918618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.912941 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/24aaf415-4614-4e59-87db-71cac71da3d9-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:58.918618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.912966 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/24aaf415-4614-4e59-87db-71cac71da3d9-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:58.918618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.912980 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rrr6d\" (UniqueName: \"kubernetes.io/projected/24aaf415-4614-4e59-87db-71cac71da3d9-kube-api-access-rrr6d\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:58.918618 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:27:58.913069 5232 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/24aaf415-4614-4e59-87db-71cac71da3d9/volumes/kubernetes.io~downward-api/istio-podinfo Oct 30 18:27:58.918618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.913084 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/24aaf415-4614-4e59-87db-71cac71da3d9-istio-podinfo" pod "24aaf415-4614-4e59-87db-71cac71da3d9" (UID: "24aaf415-4614-4e59-87db-71cac71da3d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:27:58.918618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.913312 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:58.983002 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.982648 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:27:58.983002 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.982711 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" podUID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 18:27:58.993535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:58.989978 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:27:59.040534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.016949 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/24aaf415-4614-4e59-87db-71cac71da3d9-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:27:59.041240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.041209 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:27:59.055846 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.047650 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 18:27:59.055846 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.047724 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" podUID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 18:27:59.146653 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.146603 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:27:59.206014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.205916 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": context deadline exceeded" Oct 30 18:27:59.418534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.396710 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:59.437534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.407217 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/health\": context deadline exceeded" Oct 30 18:27:59.481593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.438806 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr"] Oct 30 18:27:59.522748 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.508646 5232 cpu_manager.go:414] "containerMap: removing container" podUID="24aaf415-4614-4e59-87db-71cac71da3d9" containerName="istio-proxy" Oct 30 18:27:59.538663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.538612 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24aaf415-4614-4e59-87db-71cac71da3d9" containerName="istio-proxy" Oct 30 18:27:59.539074 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.539054 5232 cpu_manager.go:414] "containerMap: removing container" podUID="809fec18-40ef-42e0-9914-ff3b301068c0" containerName="payload-processing" Oct 30 18:27:59.539229 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.539215 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="809fec18-40ef-42e0-9914-ff3b301068c0" containerName="payload-processing" Oct 30 18:27:59.539391 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.539380 5232 cpu_manager.go:414] "containerMap: removing container" podUID="6cb32688-9676-4e30-8b67-61d6344b4d8d" containerName="revoke-keys" Oct 30 18:27:59.539496 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.539485 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cb32688-9676-4e30-8b67-61d6344b4d8d" containerName="revoke-keys" Oct 30 18:27:59.539673 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.539651 5232 cpu_manager.go:414] "containerMap: removing container" podUID="fc0db16f-42fd-4d0f-8549-33c9d7511b3d" containerName="payload-pre-processing" Oct 30 18:27:59.539776 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.539763 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc0db16f-42fd-4d0f-8549-33c9d7511b3d" containerName="payload-pre-processing" Oct 30 18:27:59.540187 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.540172 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7dbbd730-9357-4f2f-b2d3-f41c877205e3" containerName="istio-proxy" Oct 30 18:27:59.540296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.540284 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7dbbd730-9357-4f2f-b2d3-f41c877205e3" containerName="istio-proxy" Oct 30 18:27:59.540769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.540737 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="809fec18-40ef-42e0-9914-ff3b301068c0" containerName="payload-processing" Oct 30 18:27:59.540918 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.540907 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cb32688-9676-4e30-8b67-61d6344b4d8d" containerName="revoke-keys" Oct 30 18:27:59.541032 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.541020 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc0db16f-42fd-4d0f-8549-33c9d7511b3d" containerName="payload-pre-processing" Oct 30 18:27:59.541139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.541129 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="24aaf415-4614-4e59-87db-71cac71da3d9" containerName="istio-proxy" Oct 30 18:27:59.579560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.576859 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" Oct 30 18:27:59.650255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.637872 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:59.650255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.638919 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:27:59.668648 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.660012 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-69c44c-dockercfg-lxfbm\"" Oct 30 18:27:59.668648 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.660191 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-69c44c\"" Oct 30 18:27:59.729536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.726644 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:59.729536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.727267 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:27:59.759480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.759431 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": context deadline exceeded" Oct 30 18:27:59.775546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.760450 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:27:59.947705 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.947670 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr"] Oct 30 18:27:59.947973 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.804142 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lzksj\" (UniqueName: \"kubernetes.io/projected/061ebd64-fd21-4ff1-ae1e-27148f48aa3c-kube-api-access-lzksj\") pod \"payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr\" (UID: \"061ebd64-fd21-4ff1-ae1e-27148f48aa3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" Oct 30 18:27:59.948167 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.948149 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061ebd64-fd21-4ff1-ae1e-27148f48aa3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr\" (UID: \"061ebd64-fd21-4ff1-ae1e-27148f48aa3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" Oct 30 18:27:59.948483 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.912796 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:59.957906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.802672 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:59.959335 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.959308 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:27:59.960304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.917608 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:59.960719 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.804224 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:27:59.973480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.967642 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:27:59.973480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.971176 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/health\": context deadline exceeded" Oct 30 18:27:59.991749 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.991710 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z" event={"ID":"24aaf415-4614-4e59-87db-71cac71da3d9","Type":"ContainerDied","Data":"a2fc79d7364d5fae04cdc807760f63722c36728f157220b9569db75e6dd4063e"} Oct 30 18:27:59.992481 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.992391 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:27:59.992702 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.992680 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" podUID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 18:27:59.993133 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.992430 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:00.005537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:27:59.994821 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:28:00.043597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.024429 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6"] Oct 30 18:28:00.043597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.040222 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/health\": context deadline exceeded" Oct 30 18:28:00.045300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.045272 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 18:28:00.045522 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.045483 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" podUID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 18:28:00.077533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.053377 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lzksj\" (UniqueName: \"kubernetes.io/projected/061ebd64-fd21-4ff1-ae1e-27148f48aa3c-kube-api-access-lzksj\") pod \"payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr\" (UID: \"061ebd64-fd21-4ff1-ae1e-27148f48aa3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" Oct 30 18:28:00.077946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.077912 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061ebd64-fd21-4ff1-ae1e-27148f48aa3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr\" (UID: \"061ebd64-fd21-4ff1-ae1e-27148f48aa3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" Oct 30 18:28:00.104535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.079810 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061ebd64-fd21-4ff1-ae1e-27148f48aa3c-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr\" (UID: \"061ebd64-fd21-4ff1-ae1e-27148f48aa3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" Oct 30 18:28:00.133535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.128767 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:00.133535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.128926 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6" Oct 30 18:28:00.148199 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.148158 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:00.204533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.191459 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lzksj\" (UniqueName: \"kubernetes.io/projected/061ebd64-fd21-4ff1-ae1e-27148f48aa3c-kube-api-access-lzksj\") pod \"payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr\" (UID: \"061ebd64-fd21-4ff1-ae1e-27148f48aa3c\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" Oct 30 18:28:00.204533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.192342 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:28:00.221895 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.221618 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" Oct 30 18:28:00.352657 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.223062 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dr4dm\" (UniqueName: \"kubernetes.io/projected/52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76-kube-api-access-dr4dm\") pod \"payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6\" (UID: \"52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76\") " pod="openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6" Oct 30 18:28:00.352657 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.223307 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6\" (UID: \"52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76\") " pod="openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6" Oct 30 18:28:00.352657 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.254774 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:00.352657 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.270897 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:28:00.352657 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.326799 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6"] Oct 30 18:28:00.354607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.354546 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z"] Oct 30 18:28:00.364534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.355961 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dr4dm\" (UniqueName: \"kubernetes.io/projected/52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76-kube-api-access-dr4dm\") pod \"payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6\" (UID: \"52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76\") " pod="openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6" Oct 30 18:28:00.364534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.356203 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6\" (UID: \"52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76\") " pod="openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6" Oct 30 18:28:00.373535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.365693 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6\" (UID: \"52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76\") " pod="openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6" Oct 30 18:28:00.560970 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.534201 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dr4dm\" (UniqueName: \"kubernetes.io/projected/52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76-kube-api-access-dr4dm\") pod \"payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6\" (UID: \"52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76\") " pod="openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6" Oct 30 18:28:00.660711 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.656754 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:00.660711 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.657823 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:28:00.700629 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.669766 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": context deadline exceeded" Oct 30 18:28:00.700629 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.671846 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:28:00.705102 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.705069 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6" Oct 30 18:28:00.811047 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.739658 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:00.811047 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.764598 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:00.962601 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:00.960215 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" probeResult="failure" output="Get \"https://10.133.0.74:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:01.080306 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:01.012699 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:28:01.080306 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:01.077437 5232 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/health\": context deadline exceeded" Oct 30 18:28:01.107567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:01.082482 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded" Oct 30 18:28:01.197640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:01.099181 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 18:28:01.197640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:01.195714 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" podUID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 18:28:01.197640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:01.112174 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:01.197640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:01.110279 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:28:01.197640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:01.112131 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:28:01.197640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:01.196232 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" podUID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 18:28:01.197640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:01.193991 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:28:01.281375 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:01.264703 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:01.706777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:01.706641 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded" Oct 30 18:28:01.821292 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:01.819867 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:01.845278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:01.845227 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:01.937532 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:28:01.919005 5232 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.119s" Oct 30 18:28:02.038181 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.035368 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:28:02.038181 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.035412 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-h9v4z"] Oct 30 18:28:02.038181 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.035467 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:28:02.038181 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.035548 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:28:02.147538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.139898 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:28:02.176683 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:28:02.176651 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod061ebd64_fd21_4ff1_ae1e_27148f48aa3c.slice/crio-9abbf59f86a1a685e8fce04cc080bee849a7ae526d233b7b55e2c5f0f1bfe421 WatchSource:0}: Error finding container 9abbf59f86a1a685e8fce04cc080bee849a7ae526d233b7b55e2c5f0f1bfe421: Status 404 returned error can't find the container with id 9abbf59f86a1a685e8fce04cc080bee849a7ae526d233b7b55e2c5f0f1bfe421 Oct 30 18:28:02.206617 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.206571 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:02.272673 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.272620 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" probeResult="failure" output="Get \"https://10.133.0.90:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:02.295595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.203706 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:02.306693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.306648 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" probeResult="failure" output="Get \"https://10.133.0.95:8000/ready\": context deadline exceeded" Oct 30 18:28:02.333670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.308579 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": context deadline exceeded" Oct 30 18:28:02.371289 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.371254 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:28:02.371642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.371611 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" podUID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 18:28:02.394684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.393372 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb"] Oct 30 18:28:02.394684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.393641 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" podUID="43d92d4d-f62f-4b68-a9bf-89186482a244" containerName="istio-proxy" containerID="cri-o://95c4a97f1c15605ffa91358c2d90085d5d961e13c5251c928309c0b4fda6ddbd" gracePeriod=30 Oct 30 18:28:02.394684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.393859 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": context deadline exceeded" Oct 30 18:28:02.407660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.407615 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:02.478007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.477765 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 18:28:02.478318 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.478271 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" podUID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 18:28:02.936546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.935071 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" probeResult="failure" output="Get \"https://10.133.0.44:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:02.976925 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.976821 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" probeResult="failure" output="Get \"https://10.133.0.81:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:03.052565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:02.999795 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" probeResult="failure" output="Get \"https://10.133.0.43:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:03.132584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.086441 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 18:28:03.132920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.132876 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" podUID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 18:28:03.155833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.155792 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:28:03.156111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.156063 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" podUID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 18:28:03.425545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.403553 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:03.425545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.406642 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" probeResult="failure" output="Get \"https://10.133.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:03.437620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.436840 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" probeResult="failure" output="Get \"https://10.133.0.93:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:28:03.942713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.925546 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24aaf415-4614-4e59-87db-71cac71da3d9" path="/var/lib/kubelet/pods/24aaf415-4614-4e59-87db-71cac71da3d9/volumes" Oct 30 18:28:03.942713 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:28:03.926031 5232 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.12s" Oct 30 18:28:03.942713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.926095 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:28:03.942713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.926127 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr"] Oct 30 18:28:03.942713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.926190 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:28:03.942713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.926208 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" event={"ID":"061ebd64-fd21-4ff1-ae1e-27148f48aa3c","Type":"ContainerStarted","Data":"9abbf59f86a1a685e8fce04cc080bee849a7ae526d233b7b55e2c5f0f1bfe421"} Oct 30 18:28:03.942713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.926278 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:28:03.942713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.926319 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:28:03.942713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.926348 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:28:03.942713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.926380 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:28:03.942713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.926410 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:28:03.942713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.926427 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg"] Oct 30 18:28:03.981845 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.969424 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:28:03.981845 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:03.969501 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" podUID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 18:28:04.024049 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.024008 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg"] Oct 30 18:28:04.024336 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.024320 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:28:04.024608 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.024592 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:28:04.038530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.025763 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg" Oct 30 18:28:04.045466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.045436 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 18:28:04.072872 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.072828 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" podUID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 18:28:04.078623 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.072073 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:28:04.175546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.072014 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:28:04.177682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.177648 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d151fce2-732c-46f2-9348-0c8889e1f000-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg\" (UID: \"d151fce2-732c-46f2-9348-0c8889e1f000\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg" Oct 30 18:28:04.204536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.204443 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d151fce2-732c-46f2-9348-0c8889e1f000-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg\" (UID: \"d151fce2-732c-46f2-9348-0c8889e1f000\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg" Oct 30 18:28:04.204905 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.204883 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xfq9t\" (UniqueName: \"kubernetes.io/projected/d151fce2-732c-46f2-9348-0c8889e1f000-kube-api-access-xfq9t\") pod \"maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg\" (UID: \"d151fce2-732c-46f2-9348-0c8889e1f000\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg" Oct 30 18:28:04.287727 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.208397 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6"] Oct 30 18:28:04.292541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.292493 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6" event={"ID":"52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76","Type":"ContainerStarted","Data":"0bd2c106c548e289c8edfe4b5a8378824967c8f039731bbc88da3bdcab08a5b0"} Oct 30 18:28:04.387184 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.387153 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d151fce2-732c-46f2-9348-0c8889e1f000-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg\" (UID: \"d151fce2-732c-46f2-9348-0c8889e1f000\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg" Oct 30 18:28:04.388119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.388101 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d151fce2-732c-46f2-9348-0c8889e1f000-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg\" (UID: \"d151fce2-732c-46f2-9348-0c8889e1f000\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg" Oct 30 18:28:04.388385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.388365 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xfq9t\" (UniqueName: \"kubernetes.io/projected/d151fce2-732c-46f2-9348-0c8889e1f000-kube-api-access-xfq9t\") pod \"maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg\" (UID: \"d151fce2-732c-46f2-9348-0c8889e1f000\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg" Oct 30 18:28:04.388911 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.388057 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d151fce2-732c-46f2-9348-0c8889e1f000-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg\" (UID: \"d151fce2-732c-46f2-9348-0c8889e1f000\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg" Oct 30 18:28:04.391643 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.391616 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d151fce2-732c-46f2-9348-0c8889e1f000-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg\" (UID: \"d151fce2-732c-46f2-9348-0c8889e1f000\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg" Oct 30 18:28:04.475207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.475093 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xfq9t\" (UniqueName: \"kubernetes.io/projected/d151fce2-732c-46f2-9348-0c8889e1f000-kube-api-access-xfq9t\") pod \"maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg\" (UID: \"d151fce2-732c-46f2-9348-0c8889e1f000\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg" Oct 30 18:28:04.561900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.561867 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg" Oct 30 18:28:04.874067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.874025 5232 patch_prober.go:29] interesting pod/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:28:04.874357 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.874094 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" podUID="43d92d4d-f62f-4b68-a9bf-89186482a244" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:28:04.967724 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.967692 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:28:04.967978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.967948 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" podUID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 18:28:04.968258 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:28:04.968239 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd151fce2_732c_46f2_9348_0c8889e1f000.slice/crio-74837c902a57e6020cb29d044b45f343c2a6227c5dbc3c2b8efe2baca2365009 WatchSource:0}: Error finding container 74837c902a57e6020cb29d044b45f343c2a6227c5dbc3c2b8efe2baca2365009: Status 404 returned error can't find the container with id 74837c902a57e6020cb29d044b45f343c2a6227c5dbc3c2b8efe2baca2365009 Oct 30 18:28:04.969817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:04.969790 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg"] Oct 30 18:28:05.069564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:05.058836 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 18:28:05.069564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:05.058916 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" podUID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 18:28:05.364209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:05.364144 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" event={"ID":"061ebd64-fd21-4ff1-ae1e-27148f48aa3c","Type":"ContainerStarted","Data":"80e44c432d6101a22e83799067f1988c52addc860aabea506dd1ab62f34fbe0c"} Oct 30 18:28:05.368641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:05.368609 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" Oct 30 18:28:05.374544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:05.374491 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6" event={"ID":"52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76","Type":"ContainerStarted","Data":"dbd4403a7273ca1f579da876c0d52e60374cda59fa29f8775328e76c3ff47cc7"} Oct 30 18:28:05.375806 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:05.375786 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6" Oct 30 18:28:05.378836 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:05.378811 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg" event={"ID":"d151fce2-732c-46f2-9348-0c8889e1f000","Type":"ContainerStarted","Data":"74837c902a57e6020cb29d044b45f343c2a6227c5dbc3c2b8efe2baca2365009"} Oct 30 18:28:05.431606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:05.429102 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" podStartSLOduration=6.429077733 podStartE2EDuration="6.429077733s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:05.409268048 +0000 UTC m=+1584.725688372" watchObservedRunningTime="2026-10-30 18:28:05.429077733 +0000 UTC m=+1584.745498056" Oct 30 18:28:05.495109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:05.495049 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6" podStartSLOduration=6.495031879 podStartE2EDuration="6.495031879s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:27:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:05.490605756 +0000 UTC m=+1584.807026078" watchObservedRunningTime="2026-10-30 18:28:05.495031879 +0000 UTC m=+1584.811452201" Oct 30 18:28:05.987531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:05.987397 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" start-of-body= Oct 30 18:28:05.987531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:05.987465 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" podUID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.160:15021/healthz/ready\": dial tcp 10.133.0.160:15021: connect: connection refused" Oct 30 18:28:06.049540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.048884 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" start-of-body= Oct 30 18:28:06.049540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.048942 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" podUID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.161:15021/healthz/ready\": dial tcp 10.133.0.161:15021: connect: connection refused" Oct 30 18:28:06.389371 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.389279 5232 generic.go:356] "Generic (PLEG): container finished" podID="d151fce2-732c-46f2-9348-0c8889e1f000" containerID="1074582ce2c8e166e3c40d9f09200b727bce7809c4fd18579e5843257480fabf" exitCode=0 Oct 30 18:28:06.389585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.389364 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg" event={"ID":"d151fce2-732c-46f2-9348-0c8889e1f000","Type":"ContainerDied","Data":"1074582ce2c8e166e3c40d9f09200b727bce7809c4fd18579e5843257480fabf"} Oct 30 18:28:06.791802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.791765 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w"] Oct 30 18:28:06.795147 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.795124 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.830121 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.830071 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w"] Oct 30 18:28:06.885766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.885727 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sk7rc\" (UniqueName: \"kubernetes.io/projected/703d9be7-1970-4a72-8a98-7c15fa6108f1-kube-api-access-sk7rc\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.885931 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.885771 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.885931 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.885912 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.886027 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.885944 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.886071 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.886055 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.886172 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.886160 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.886212 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.886183 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.886263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.886251 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.886298 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.886277 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/703d9be7-1970-4a72-8a98-7c15fa6108f1-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.970697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.970667 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:28:06.971243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.971216 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:28:06.972066 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.972041 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:28:06.988401 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.988367 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sk7rc\" (UniqueName: \"kubernetes.io/projected/703d9be7-1970-4a72-8a98-7c15fa6108f1-kube-api-access-sk7rc\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.988401 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.988410 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.988748 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.988520 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.988748 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.988568 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.988748 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.988663 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.988939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.988778 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.988939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.988809 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.988939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.988882 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.988939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.988919 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/703d9be7-1970-4a72-8a98-7c15fa6108f1-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.989171 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.988937 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-workload-certs\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.989659 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.989622 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-workload-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.990269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.990225 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-data\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.990372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.990338 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-credential-socket\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.990441 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.990417 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/703d9be7-1970-4a72-8a98-7c15fa6108f1-istiod-ca-cert\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.991646 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.991624 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-envoy\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.992718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.992699 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-podinfo\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:06.999725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:06.999700 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7"] Oct 30 18:28:07.024250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.022946 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sk7rc\" (UniqueName: \"kubernetes.io/projected/703d9be7-1970-4a72-8a98-7c15fa6108f1-kube-api-access-sk7rc\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:07.025786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.025752 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-token\") pod \"e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:07.046527 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.046440 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:28:07.046930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.046905 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:28:07.047995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.047971 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:28:07.117880 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.117844 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:07.475283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.475234 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w"] Oct 30 18:28:07.475815 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:28:07.475784 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod703d9be7_1970_4a72_8a98_7c15fa6108f1.slice/crio-fcea701bb3c0066f2e862841c21ba3c8163d8b65a8246737c245e41d6dab1e24 WatchSource:0}: Error finding container fcea701bb3c0066f2e862841c21ba3c8163d8b65a8246737c245e41d6dab1e24: Status 404 returned error can't find the container with id fcea701bb3c0066f2e862841c21ba3c8163d8b65a8246737c245e41d6dab1e24 Oct 30 18:28:07.480304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.480259 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:28:07.480420 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.480357 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:28:07.480420 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.480403 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:28:07.572729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.572704 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg" Oct 30 18:28:07.717075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.717034 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d151fce2-732c-46f2-9348-0c8889e1f000-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d151fce2-732c-46f2-9348-0c8889e1f000-maas-api-openshift-ca\") pod \"d151fce2-732c-46f2-9348-0c8889e1f000\" (UID: \"d151fce2-732c-46f2-9348-0c8889e1f000\") " Oct 30 18:28:07.717260 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.717082 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d151fce2-732c-46f2-9348-0c8889e1f000-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d151fce2-732c-46f2-9348-0c8889e1f000-maas-api-xks-ca\") pod \"d151fce2-732c-46f2-9348-0c8889e1f000\" (UID: \"d151fce2-732c-46f2-9348-0c8889e1f000\") " Oct 30 18:28:07.717321 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.717305 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d151fce2-732c-46f2-9348-0c8889e1f000-kube-api-access-xfq9t\" (UniqueName: \"kubernetes.io/projected/d151fce2-732c-46f2-9348-0c8889e1f000-kube-api-access-xfq9t\") pod \"d151fce2-732c-46f2-9348-0c8889e1f000\" (UID: \"d151fce2-732c-46f2-9348-0c8889e1f000\") " Oct 30 18:28:07.718152 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.717792 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d151fce2-732c-46f2-9348-0c8889e1f000-maas-api-openshift-ca" pod "d151fce2-732c-46f2-9348-0c8889e1f000" (UID: "d151fce2-732c-46f2-9348-0c8889e1f000"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:07.718152 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.718071 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d151fce2-732c-46f2-9348-0c8889e1f000-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:07.720336 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.720312 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d151fce2-732c-46f2-9348-0c8889e1f000-maas-api-xks-ca" pod "d151fce2-732c-46f2-9348-0c8889e1f000" (UID: "d151fce2-732c-46f2-9348-0c8889e1f000"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:07.720804 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.720784 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d151fce2-732c-46f2-9348-0c8889e1f000-kube-api-access-xfq9t" pod "d151fce2-732c-46f2-9348-0c8889e1f000" (UID: "d151fce2-732c-46f2-9348-0c8889e1f000"). InnerVolumeSpecName "kube-api-access-xfq9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:07.824598 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.823036 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d151fce2-732c-46f2-9348-0c8889e1f000-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:07.824598 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.823088 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xfq9t\" (UniqueName: \"kubernetes.io/projected/d151fce2-732c-46f2-9348-0c8889e1f000-kube-api-access-xfq9t\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:07.911005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:07.910938 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:28:08.025062 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.025034 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-podinfo\") pod \"43d92d4d-f62f-4b68-a9bf-89186482a244\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " Oct 30 18:28:08.025236 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.025080 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-token\" (UniqueName: \"kubernetes.io/projected/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-token\") pod \"43d92d4d-f62f-4b68-a9bf-89186482a244\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " Oct 30 18:28:08.025236 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.025156 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/43d92d4d-f62f-4b68-a9bf-89186482a244-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43d92d4d-f62f-4b68-a9bf-89186482a244-istiod-ca-cert\") pod \"43d92d4d-f62f-4b68-a9bf-89186482a244\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " Oct 30 18:28:08.025236 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.025227 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/43d92d4d-f62f-4b68-a9bf-89186482a244-kube-api-access-btsvg\" (UniqueName: \"kubernetes.io/projected/43d92d4d-f62f-4b68-a9bf-89186482a244-kube-api-access-btsvg\") pod \"43d92d4d-f62f-4b68-a9bf-89186482a244\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " Oct 30 18:28:08.025540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.025284 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-workload-certs\") pod \"43d92d4d-f62f-4b68-a9bf-89186482a244\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " Oct 30 18:28:08.025540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.025397 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-workload-socket\") pod \"43d92d4d-f62f-4b68-a9bf-89186482a244\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " Oct 30 18:28:08.025540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.025441 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-envoy\") pod \"43d92d4d-f62f-4b68-a9bf-89186482a244\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " Oct 30 18:28:08.025540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.025490 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-data\") pod \"43d92d4d-f62f-4b68-a9bf-89186482a244\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " Oct 30 18:28:08.025773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.025682 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-credential-socket\") pod \"43d92d4d-f62f-4b68-a9bf-89186482a244\" (UID: \"43d92d4d-f62f-4b68-a9bf-89186482a244\") " Oct 30 18:28:08.026697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.025998 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-workload-certs" pod "43d92d4d-f62f-4b68-a9bf-89186482a244" (UID: "43d92d4d-f62f-4b68-a9bf-89186482a244"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:08.026697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.026083 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-credential-socket" pod "43d92d4d-f62f-4b68-a9bf-89186482a244" (UID: "43d92d4d-f62f-4b68-a9bf-89186482a244"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:08.026697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.026488 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:08.026697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.026535 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:08.026697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.026608 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43d92d4d-f62f-4b68-a9bf-89186482a244-istiod-ca-cert" pod "43d92d4d-f62f-4b68-a9bf-89186482a244" (UID: "43d92d4d-f62f-4b68-a9bf-89186482a244"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:08.027252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.026768 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-workload-socket" pod "43d92d4d-f62f-4b68-a9bf-89186482a244" (UID: "43d92d4d-f62f-4b68-a9bf-89186482a244"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:08.027252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.026825 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-data" pod "43d92d4d-f62f-4b68-a9bf-89186482a244" (UID: "43d92d4d-f62f-4b68-a9bf-89186482a244"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:08.028467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.028442 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-token" pod "43d92d4d-f62f-4b68-a9bf-89186482a244" (UID: "43d92d4d-f62f-4b68-a9bf-89186482a244"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:08.028632 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.028607 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-envoy" pod "43d92d4d-f62f-4b68-a9bf-89186482a244" (UID: "43d92d4d-f62f-4b68-a9bf-89186482a244"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:08.028802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.028766 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-podinfo" pod "43d92d4d-f62f-4b68-a9bf-89186482a244" (UID: "43d92d4d-f62f-4b68-a9bf-89186482a244"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:08.029254 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.029226 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43d92d4d-f62f-4b68-a9bf-89186482a244-kube-api-access-btsvg" pod "43d92d4d-f62f-4b68-a9bf-89186482a244" (UID: "43d92d4d-f62f-4b68-a9bf-89186482a244"). InnerVolumeSpecName "kube-api-access-btsvg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:08.127903 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.127859 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:08.127903 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.127902 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:08.128119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.127917 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:08.128119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.127930 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:08.128119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.127942 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/43d92d4d-f62f-4b68-a9bf-89186482a244-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:08.128119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.127954 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/43d92d4d-f62f-4b68-a9bf-89186482a244-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:08.128119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.127967 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btsvg\" (UniqueName: \"kubernetes.io/projected/43d92d4d-f62f-4b68-a9bf-89186482a244-kube-api-access-btsvg\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:08.410580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.410543 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" event={"ID":"703d9be7-1970-4a72-8a98-7c15fa6108f1","Type":"ContainerStarted","Data":"4b60ebcd19d90f90816430447fb279f0960a6fed926fc8d2af989dc1cbee3672"} Oct 30 18:28:08.410815 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.410619 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" event={"ID":"703d9be7-1970-4a72-8a98-7c15fa6108f1","Type":"ContainerStarted","Data":"fcea701bb3c0066f2e862841c21ba3c8163d8b65a8246737c245e41d6dab1e24"} Oct 30 18:28:08.412490 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.412460 5232 generic.go:356] "Generic (PLEG): container finished" podID="43d92d4d-f62f-4b68-a9bf-89186482a244" containerID="95c4a97f1c15605ffa91358c2d90085d5d961e13c5251c928309c0b4fda6ddbd" exitCode=0 Oct 30 18:28:08.412620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.412520 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" Oct 30 18:28:08.412620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.412546 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" event={"ID":"43d92d4d-f62f-4b68-a9bf-89186482a244","Type":"ContainerDied","Data":"95c4a97f1c15605ffa91358c2d90085d5d961e13c5251c928309c0b4fda6ddbd"} Oct 30 18:28:08.412620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.412596 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb" event={"ID":"43d92d4d-f62f-4b68-a9bf-89186482a244","Type":"ContainerDied","Data":"07dff41b6e41cc391ad7f9b01afc715f517733acacc2bdc8448816b2f7688605"} Oct 30 18:28:08.412620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.412617 5232 scope.go:122] "RemoveContainer" containerID="95c4a97f1c15605ffa91358c2d90085d5d961e13c5251c928309c0b4fda6ddbd" Oct 30 18:28:08.414265 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.414232 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg" Oct 30 18:28:08.414371 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.414245 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg" event={"ID":"d151fce2-732c-46f2-9348-0c8889e1f000","Type":"ContainerDied","Data":"74837c902a57e6020cb29d044b45f343c2a6227c5dbc3c2b8efe2baca2365009"} Oct 30 18:28:08.414371 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.414359 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="74837c902a57e6020cb29d044b45f343c2a6227c5dbc3c2b8efe2baca2365009" Oct 30 18:28:08.428800 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.428778 5232 scope.go:122] "RemoveContainer" containerID="13d441f3a5c46ec1e0f3f3e38fc0f8692d40a5ce5bcd6d95ba64eeb47729137d" Oct 30 18:28:08.439078 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.439055 5232 scope.go:122] "RemoveContainer" containerID="95c4a97f1c15605ffa91358c2d90085d5d961e13c5251c928309c0b4fda6ddbd" Oct 30 18:28:08.439362 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:28:08.439337 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95c4a97f1c15605ffa91358c2d90085d5d961e13c5251c928309c0b4fda6ddbd\": container with ID starting with 95c4a97f1c15605ffa91358c2d90085d5d961e13c5251c928309c0b4fda6ddbd not found: ID does not exist" containerID="95c4a97f1c15605ffa91358c2d90085d5d961e13c5251c928309c0b4fda6ddbd" Oct 30 18:28:08.439452 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.439371 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95c4a97f1c15605ffa91358c2d90085d5d961e13c5251c928309c0b4fda6ddbd"} err="failed to get container status \"95c4a97f1c15605ffa91358c2d90085d5d961e13c5251c928309c0b4fda6ddbd\": rpc error: code = NotFound desc = could not find container \"95c4a97f1c15605ffa91358c2d90085d5d961e13c5251c928309c0b4fda6ddbd\": container with ID starting with 95c4a97f1c15605ffa91358c2d90085d5d961e13c5251c928309c0b4fda6ddbd not found: ID does not exist" Oct 30 18:28:08.439452 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.439388 5232 scope.go:122] "RemoveContainer" containerID="13d441f3a5c46ec1e0f3f3e38fc0f8692d40a5ce5bcd6d95ba64eeb47729137d" Oct 30 18:28:08.439729 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:28:08.439710 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13d441f3a5c46ec1e0f3f3e38fc0f8692d40a5ce5bcd6d95ba64eeb47729137d\": container with ID starting with 13d441f3a5c46ec1e0f3f3e38fc0f8692d40a5ce5bcd6d95ba64eeb47729137d not found: ID does not exist" containerID="13d441f3a5c46ec1e0f3f3e38fc0f8692d40a5ce5bcd6d95ba64eeb47729137d" Oct 30 18:28:08.439799 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.439734 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13d441f3a5c46ec1e0f3f3e38fc0f8692d40a5ce5bcd6d95ba64eeb47729137d"} err="failed to get container status \"13d441f3a5c46ec1e0f3f3e38fc0f8692d40a5ce5bcd6d95ba64eeb47729137d\": rpc error: code = NotFound desc = could not find container \"13d441f3a5c46ec1e0f3f3e38fc0f8692d40a5ce5bcd6d95ba64eeb47729137d\": container with ID starting with 13d441f3a5c46ec1e0f3f3e38fc0f8692d40a5ce5bcd6d95ba64eeb47729137d not found: ID does not exist" Oct 30 18:28:08.454867 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.454827 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" podStartSLOduration=2.45481382 podStartE2EDuration="2.45481382s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:08.452080506 +0000 UTC m=+1587.768500826" watchObservedRunningTime="2026-10-30 18:28:08.45481382 +0000 UTC m=+1587.771234205" Oct 30 18:28:08.510619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.510580 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb"] Oct 30 18:28:08.514496 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.514477 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-550276-openshift-default-5496d6dbc7-45knb"] Oct 30 18:28:08.805045 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:08.805012 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="43d92d4d-f62f-4b68-a9bf-89186482a244" path="/var/lib/kubelet/pods/43d92d4d-f62f-4b68-a9bf-89186482a244/volumes" Oct 30 18:28:09.122091 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:09.122009 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:09.141753 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:09.141711 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 18:28:09.141982 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:09.141786 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" podUID="703d9be7-1970-4a72-8a98-7c15fa6108f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 18:28:09.770338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:09.769434 5232 patch_prober.go:29] interesting pod/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": read tcp 10.133.0.2:37382->10.133.0.198:15021: read: connection reset by peer" start-of-body= Oct 30 18:28:09.770338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:09.769562 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" podUID="ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": read tcp 10.133.0.2:37382->10.133.0.198:15021: read: connection reset by peer" Oct 30 18:28:10.134739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:10.134642 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 18:28:10.134739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:10.134710 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" podUID="703d9be7-1970-4a72-8a98-7c15fa6108f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 18:28:10.456880 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:10.456771 5232 generic.go:356] "Generic (PLEG): container finished" podID="ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" containerID="78057977b8c711a6e9d6bdf34ef949a035e90cf4113d1777a0d5f0e401cd9865" exitCode=0 Oct 30 18:28:10.456880 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:10.456851 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" event={"ID":"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac","Type":"ContainerDied","Data":"78057977b8c711a6e9d6bdf34ef949a035e90cf4113d1777a0d5f0e401cd9865"} Oct 30 18:28:10.457609 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:10.457590 5232 scope.go:122] "RemoveContainer" containerID="78057977b8c711a6e9d6bdf34ef949a035e90cf4113d1777a0d5f0e401cd9865" Oct 30 18:28:10.459127 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:10.459095 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:28:10.459206 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:10.459167 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:28:10.459206 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:10.459198 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:28:11.127533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:11.126997 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 18:28:11.127533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:11.127067 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" podUID="703d9be7-1970-4a72-8a98-7c15fa6108f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 18:28:11.503165 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:11.503132 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" event={"ID":"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac","Type":"ContainerStarted","Data":"e3c5bd05ecc786eef012e9a04c6197c94ea48748f81a7a89ab6a74820f34a2d7"} Oct 30 18:28:11.885711 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:11.885609 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk"] Oct 30 18:28:11.890157 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:11.890110 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-ae3373-757hk"] Oct 30 18:28:12.128961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:12.128919 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 18:28:12.129377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:12.129343 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" podUID="703d9be7-1970-4a72-8a98-7c15fa6108f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 18:28:12.257087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:12.257047 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:28:12.258735 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:12.258686 5232 patch_prober.go:29] interesting pod/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 18:28:12.258907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:12.258778 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" podUID="ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 18:28:12.805590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:12.805559 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af1b2984-d8ad-44a9-a5d9-85ea9efea63a" path="/var/lib/kubelet/pods/af1b2984-d8ad-44a9-a5d9-85ea9efea63a/volumes" Oct 30 18:28:12.987630 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:12.987597 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m"] Oct 30 18:28:13.071016 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:13.070906 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z"] Oct 30 18:28:13.173571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:13.170935 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 18:28:13.173571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:13.170994 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" podUID="703d9be7-1970-4a72-8a98-7c15fa6108f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 18:28:13.263082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:13.263048 5232 patch_prober.go:29] interesting pod/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 18:28:13.263399 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:13.263352 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" podUID="ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 18:28:14.131535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:14.125885 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" start-of-body= Oct 30 18:28:14.131535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:14.125946 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" podUID="703d9be7-1970-4a72-8a98-7c15fa6108f1" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.208:15021/healthz/ready\": dial tcp 10.133.0.208:15021: connect: connection refused" Oct 30 18:28:14.260408 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:14.259643 5232 patch_prober.go:29] interesting pod/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 18:28:14.260408 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:14.259689 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" podUID="ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 18:28:15.139011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:15.138974 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:15.139437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:15.139391 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:15.140132 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:15.140107 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:15.257402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:15.257366 5232 patch_prober.go:29] interesting pod/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 18:28:15.257755 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:15.257726 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" podUID="ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 18:28:15.879976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:15.879943 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4"] Oct 30 18:28:15.880183 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:15.880159 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" podUID="92f3b6d0-dc11-4f7b-ac74-f33d3d469d10" containerName="maas-api" containerID="cri-o://29faeca69bacad3df2b82e9587c9805604d3b8d39ce4e1689d854017f6c3a01a" gracePeriod=30 Oct 30 18:28:16.461540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.450625 5232 patch_prober.go:29] interesting pod/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 18:28:16.461540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.450686 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" podUID="ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 18:28:16.499547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.487124 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" Oct 30 18:28:16.528820 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.501723 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6" Oct 30 18:28:16.645081 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.643723 5232 generic.go:356] "Generic (PLEG): container finished" podID="92f3b6d0-dc11-4f7b-ac74-f33d3d469d10" containerID="29faeca69bacad3df2b82e9587c9805604d3b8d39ce4e1689d854017f6c3a01a" exitCode=0 Oct 30 18:28:16.645081 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.644939 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" event={"ID":"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10","Type":"ContainerDied","Data":"29faeca69bacad3df2b82e9587c9805604d3b8d39ce4e1689d854017f6c3a01a"} Oct 30 18:28:16.645081 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.644993 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" event={"ID":"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10","Type":"ContainerDied","Data":"35cdf398d2d166ef6c28e6433d019d095528a26bbc09fc8d72bd8d76faea889f"} Oct 30 18:28:16.645081 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.645019 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="35cdf398d2d166ef6c28e6433d019d095528a26bbc09fc8d72bd8d76faea889f" Oct 30 18:28:16.647274 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.646936 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" Oct 30 18:28:16.846061 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.844882 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-maas-api-tls\") pod \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\" (UID: \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\") " Oct 30 18:28:16.846061 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.844942 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-metrics-certs\" (UniqueName: \"kubernetes.io/secret/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-metrics-certs\") pod \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\" (UID: \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\") " Oct 30 18:28:16.846061 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.845018 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-openshift-service-ca\") pod \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\" (UID: \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\") " Oct 30 18:28:16.846061 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.845153 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-kube-api-access-gmj8m\" (UniqueName: \"kubernetes.io/projected/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-kube-api-access-gmj8m\") pod \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\" (UID: \"92f3b6d0-dc11-4f7b-ac74-f33d3d469d10\") " Oct 30 18:28:16.847046 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.847018 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-openshift-service-ca" pod "92f3b6d0-dc11-4f7b-ac74-f33d3d469d10" (UID: "92f3b6d0-dc11-4f7b-ac74-f33d3d469d10"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:16.848240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.848075 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-maas-api-tls" pod "92f3b6d0-dc11-4f7b-ac74-f33d3d469d10" (UID: "92f3b6d0-dc11-4f7b-ac74-f33d3d469d10"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:16.848867 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.848823 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-kube-api-access-gmj8m" pod "92f3b6d0-dc11-4f7b-ac74-f33d3d469d10" (UID: "92f3b6d0-dc11-4f7b-ac74-f33d3d469d10"). InnerVolumeSpecName "kube-api-access-gmj8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:16.849101 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.849081 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-metrics-certs" pod "92f3b6d0-dc11-4f7b-ac74-f33d3d469d10" (UID: "92f3b6d0-dc11-4f7b-ac74-f33d3d469d10"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:16.934648 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.934608 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28"] Oct 30 18:28:16.938252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.938221 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227c01-b9v28"] Oct 30 18:28:16.963535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.947774 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gmj8m\" (UniqueName: \"kubernetes.io/projected/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-kube-api-access-gmj8m\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:16.963535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.947811 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-maas-api-tls\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:16.963535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.947828 5232 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-metrics-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:16.963535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:16.947841 5232 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10-openshift-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:17.256847 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:17.256813 5232 patch_prober.go:29] interesting pod/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" start-of-body= Oct 30 18:28:17.257033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:17.256877 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" podUID="ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.198:15021/healthz/ready\": dial tcp 10.133.0.198:15021: connect: connection refused" Oct 30 18:28:17.682592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:17.681569 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4" Oct 30 18:28:17.764831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:17.764792 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4"] Oct 30 18:28:17.776887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:17.776855 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-227c01-5794c658f5-l8gx4"] Oct 30 18:28:17.999787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:17.999575 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" podUID="552542b1-fdd2-417f-88a5-614f087e6228" containerName="payload-processing" containerID="cri-o://bb8129c4fc7a57e50f8b02a91fe453f809e049c8679ed3accda424d6388d2d89" gracePeriod=25 Oct 30 18:28:18.077533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:18.073658 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" podUID="771db49d-214d-4901-8885-50e40d3d4d88" containerName="payload-pre-processing" containerID="cri-o://65835ba5475943c5565b97503b07cce5f2c3d989d2660d206d14340a1b7acf4a" gracePeriod=25 Oct 30 18:28:18.328589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:18.327821 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:28:18.350534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:18.329393 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:28:18.350534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:18.330650 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:28:18.809970 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:18.809924 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0963e410-ad61-4bae-b699-997384481b61" path="/var/lib/kubelet/pods/0963e410-ad61-4bae-b699-997384481b61/volumes" Oct 30 18:28:18.810655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:18.810631 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92f3b6d0-dc11-4f7b-ac74-f33d3d469d10" path="/var/lib/kubelet/pods/92f3b6d0-dc11-4f7b-ac74-f33d3d469d10/volumes" Oct 30 18:28:19.023534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.020638 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" podUID="552542b1-fdd2-417f-88a5-614f087e6228" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:28:19.036580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.035650 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" podUID="771db49d-214d-4901-8885-50e40d3d4d88" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:28:19.352533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.351568 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw"] Oct 30 18:28:19.352533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.351840 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" podUID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerName="istio-proxy" containerID="cri-o://fbe4c68188085bdea85053a1d780b92618cad40a8eccaf2bc2fda31b377c5341" gracePeriod=30 Oct 30 18:28:19.389171 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.389129 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w"] Oct 30 18:28:19.389401 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.389372 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" podUID="703d9be7-1970-4a72-8a98-7c15fa6108f1" containerName="istio-proxy" containerID="cri-o://4b60ebcd19d90f90816430447fb279f0960a6fed926fc8d2af989dc1cbee3672" gracePeriod=30 Oct 30 18:28:19.392416 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.392384 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24"] Oct 30 18:28:19.392670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.392626 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" podUID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerName="istio-proxy" containerID="cri-o://f2767476296f1939812236ce78d03b4f0677055fa6bdde7665176edac0a7a606" gracePeriod=30 Oct 30 18:28:19.993863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.993821 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx"] Oct 30 18:28:19.995342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.995315 5232 cpu_manager.go:414] "containerMap: removing container" podUID="43d92d4d-f62f-4b68-a9bf-89186482a244" containerName="istio-proxy" Oct 30 18:28:19.995527 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.995488 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d92d4d-f62f-4b68-a9bf-89186482a244" containerName="istio-proxy" Oct 30 18:28:19.995651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.995640 5232 cpu_manager.go:414] "containerMap: removing container" podUID="d151fce2-732c-46f2-9348-0c8889e1f000" containerName="revoke-keys" Oct 30 18:28:19.995738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.995728 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d151fce2-732c-46f2-9348-0c8889e1f000" containerName="revoke-keys" Oct 30 18:28:19.995825 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.995816 5232 cpu_manager.go:414] "containerMap: removing container" podUID="43d92d4d-f62f-4b68-a9bf-89186482a244" containerName="istio-proxy" Oct 30 18:28:19.995911 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.995901 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="43d92d4d-f62f-4b68-a9bf-89186482a244" containerName="istio-proxy" Oct 30 18:28:19.996035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.996025 5232 cpu_manager.go:414] "containerMap: removing container" podUID="92f3b6d0-dc11-4f7b-ac74-f33d3d469d10" containerName="maas-api" Oct 30 18:28:19.996112 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.996102 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92f3b6d0-dc11-4f7b-ac74-f33d3d469d10" containerName="maas-api" Oct 30 18:28:19.997584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.996434 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="92f3b6d0-dc11-4f7b-ac74-f33d3d469d10" containerName="maas-api" Oct 30 18:28:19.997734 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.997722 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d92d4d-f62f-4b68-a9bf-89186482a244" containerName="istio-proxy" Oct 30 18:28:19.997831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.997822 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="d151fce2-732c-46f2-9348-0c8889e1f000" containerName="revoke-keys" Oct 30 18:28:19.997920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:19.997911 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="43d92d4d-f62f-4b68-a9bf-89186482a244" containerName="istio-proxy" Oct 30 18:28:20.003805 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:20.003784 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx"] Oct 30 18:28:20.004636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:20.004012 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx" Oct 30 18:28:20.080537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:20.051895 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:28:20.080537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:20.051973 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:28:20.192977 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:20.192936 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4n6h8\" (UniqueName: \"kubernetes.io/projected/b85e0598-ed75-414b-a90a-667a751cd95a-kube-api-access-4n6h8\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx\" (UID: \"b85e0598-ed75-414b-a90a-667a751cd95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx" Oct 30 18:28:20.193450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:20.193426 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b85e0598-ed75-414b-a90a-667a751cd95a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx\" (UID: \"b85e0598-ed75-414b-a90a-667a751cd95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx" Oct 30 18:28:20.193683 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:20.193663 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b85e0598-ed75-414b-a90a-667a751cd95a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx\" (UID: \"b85e0598-ed75-414b-a90a-667a751cd95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx" Oct 30 18:28:20.295641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:20.295557 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b85e0598-ed75-414b-a90a-667a751cd95a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx\" (UID: \"b85e0598-ed75-414b-a90a-667a751cd95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx" Oct 30 18:28:20.295641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:20.295601 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b85e0598-ed75-414b-a90a-667a751cd95a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx\" (UID: \"b85e0598-ed75-414b-a90a-667a751cd95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx" Oct 30 18:28:20.295853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:20.295737 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4n6h8\" (UniqueName: \"kubernetes.io/projected/b85e0598-ed75-414b-a90a-667a751cd95a-kube-api-access-4n6h8\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx\" (UID: \"b85e0598-ed75-414b-a90a-667a751cd95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx" Oct 30 18:28:20.296235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:20.296207 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b85e0598-ed75-414b-a90a-667a751cd95a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx\" (UID: \"b85e0598-ed75-414b-a90a-667a751cd95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx" Oct 30 18:28:20.298249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:20.298227 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b85e0598-ed75-414b-a90a-667a751cd95a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx\" (UID: \"b85e0598-ed75-414b-a90a-667a751cd95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx" Oct 30 18:28:20.342100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:20.342064 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4n6h8\" (UniqueName: \"kubernetes.io/projected/b85e0598-ed75-414b-a90a-667a751cd95a-kube-api-access-4n6h8\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx\" (UID: \"b85e0598-ed75-414b-a90a-667a751cd95a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx" Oct 30 18:28:20.490040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:20.490003 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx" Oct 30 18:28:21.160119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:21.160058 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx"] Oct 30 18:28:21.913084 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:21.913043 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx" event={"ID":"b85e0598-ed75-414b-a90a-667a751cd95a","Type":"ContainerStarted","Data":"edbed106988a4d3f27fb19337915a3a275f66207b996e5c53128666ed40211b7"} Oct 30 18:28:21.913302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:21.913107 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx" event={"ID":"b85e0598-ed75-414b-a90a-667a751cd95a","Type":"ContainerStarted","Data":"f41c820d6122212383d252363d0913e87c879ae5b7ae0fae5ab6c8fd68de7b6b"} Oct 30 18:28:21.970859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:21.970810 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx" podStartSLOduration=2.9707969370000002 podStartE2EDuration="2.970796937s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:21.969074537 +0000 UTC m=+1601.285494858" watchObservedRunningTime="2026-10-30 18:28:21.970796937 +0000 UTC m=+1601.287217300" Oct 30 18:28:21.973936 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:21.973904 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 30 18:28:21.974076 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:21.973956 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" podUID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:28:22.042001 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.041962 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" containerID="cri-o://0e66854e85077b884fdf444a3ae0d64f02f9bcc5f5f7397a4f02dceec67afee9" gracePeriod=45 Oct 30 18:28:22.049614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.049581 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:28:22.049766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.049646 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" podUID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:28:22.320117 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.320092 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:28:22.333883 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.333846 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-tmp-dir\") pod \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " Oct 30 18:28:22.334061 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.333943 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-dshm\") pod \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " Oct 30 18:28:22.334061 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.334010 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-tls-certs\" (UniqueName: \"kubernetes.io/secret/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-tls-certs\") pod \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " Oct 30 18:28:22.334061 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.334051 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-home\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-home\") pod \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " Oct 30 18:28:22.334229 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.334082 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-tmp-dir" pod "f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" (UID: "f7f7ebe4-e24c-4cd7-aa13-978eb635e01e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:22.334229 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.334179 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-kube-api-access-lzkq9\" (UniqueName: \"kubernetes.io/projected/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-kube-api-access-lzkq9\") pod \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " Oct 30 18:28:22.334338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.334284 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-model-cache\") pod \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\" (UID: \"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e\") " Oct 30 18:28:22.334338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.334327 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-home" pod "f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" (UID: "f7f7ebe4-e24c-4cd7-aa13-978eb635e01e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:22.334604 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.334562 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-model-cache" pod "f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" (UID: "f7f7ebe4-e24c-4cd7-aa13-978eb635e01e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:22.335094 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.335072 5232 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-tmp-dir\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.335158 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.335103 5232 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-home\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.335158 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.335117 5232 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-model-cache\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.336441 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.336415 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-dshm" pod "f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" (UID: "f7f7ebe4-e24c-4cd7-aa13-978eb635e01e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:22.336575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.336490 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-tls-certs" pod "f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" (UID: "f7f7ebe4-e24c-4cd7-aa13-978eb635e01e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:22.336575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.336539 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-kube-api-access-lzkq9" pod "f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" (UID: "f7f7ebe4-e24c-4cd7-aa13-978eb635e01e"). InnerVolumeSpecName "kube-api-access-lzkq9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:22.436644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.436550 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzkq9\" (UniqueName: \"kubernetes.io/projected/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-kube-api-access-lzkq9\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.436644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.436594 5232 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-dshm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.436644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.436611 5232 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e-tls-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:22.922205 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.922162 5232 generic.go:356] "Generic (PLEG): container finished" podID="b85e0598-ed75-414b-a90a-667a751cd95a" containerID="edbed106988a4d3f27fb19337915a3a275f66207b996e5c53128666ed40211b7" exitCode=0 Oct 30 18:28:22.922413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.922242 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx" event={"ID":"b85e0598-ed75-414b-a90a-667a751cd95a","Type":"ContainerDied","Data":"edbed106988a4d3f27fb19337915a3a275f66207b996e5c53128666ed40211b7"} Oct 30 18:28:22.923687 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.923663 5232 generic.go:356] "Generic (PLEG): container finished" podID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerID="0e66854e85077b884fdf444a3ae0d64f02f9bcc5f5f7397a4f02dceec67afee9" exitCode=0 Oct 30 18:28:22.923808 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.923721 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" Oct 30 18:28:22.923808 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.923753 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" event={"ID":"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e","Type":"ContainerDied","Data":"0e66854e85077b884fdf444a3ae0d64f02f9bcc5f5f7397a4f02dceec67afee9"} Oct 30 18:28:22.923808 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.923796 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7" event={"ID":"f7f7ebe4-e24c-4cd7-aa13-978eb635e01e","Type":"ContainerDied","Data":"c152ccf12ab6f76ade57d01a44df17d91badb5c89437ea98c6c4ff7df7345116"} Oct 30 18:28:22.923808 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.923809 5232 scope.go:122] "RemoveContainer" containerID="0e66854e85077b884fdf444a3ae0d64f02f9bcc5f5f7397a4f02dceec67afee9" Oct 30 18:28:22.934853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.934826 5232 scope.go:122] "RemoveContainer" containerID="0e66854e85077b884fdf444a3ae0d64f02f9bcc5f5f7397a4f02dceec67afee9" Oct 30 18:28:22.935192 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:28:22.935170 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e66854e85077b884fdf444a3ae0d64f02f9bcc5f5f7397a4f02dceec67afee9\": container with ID starting with 0e66854e85077b884fdf444a3ae0d64f02f9bcc5f5f7397a4f02dceec67afee9 not found: ID does not exist" containerID="0e66854e85077b884fdf444a3ae0d64f02f9bcc5f5f7397a4f02dceec67afee9" Oct 30 18:28:22.935301 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:22.935199 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e66854e85077b884fdf444a3ae0d64f02f9bcc5f5f7397a4f02dceec67afee9"} err="failed to get container status \"0e66854e85077b884fdf444a3ae0d64f02f9bcc5f5f7397a4f02dceec67afee9\": rpc error: code = NotFound desc = could not find container \"0e66854e85077b884fdf444a3ae0d64f02f9bcc5f5f7397a4f02dceec67afee9\": container with ID starting with 0e66854e85077b884fdf444a3ae0d64f02f9bcc5f5f7397a4f02dceec67afee9 not found: ID does not exist" Oct 30 18:28:23.000625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:23.000591 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7"] Oct 30 18:28:23.016176 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:23.016138 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-ef13cc/distinct-2-w6-ef13cc-kserve-6d6f7b599c-wjfh7"] Oct 30 18:28:24.108259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.108234 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx" Oct 30 18:28:24.170625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.170593 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b85e0598-ed75-414b-a90a-667a751cd95a-kube-api-access-4n6h8\" (UniqueName: \"kubernetes.io/projected/b85e0598-ed75-414b-a90a-667a751cd95a-kube-api-access-4n6h8\") pod \"b85e0598-ed75-414b-a90a-667a751cd95a\" (UID: \"b85e0598-ed75-414b-a90a-667a751cd95a\") " Oct 30 18:28:24.170785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.170681 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b85e0598-ed75-414b-a90a-667a751cd95a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b85e0598-ed75-414b-a90a-667a751cd95a-maas-api-openshift-ca\") pod \"b85e0598-ed75-414b-a90a-667a751cd95a\" (UID: \"b85e0598-ed75-414b-a90a-667a751cd95a\") " Oct 30 18:28:24.170842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.170818 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b85e0598-ed75-414b-a90a-667a751cd95a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b85e0598-ed75-414b-a90a-667a751cd95a-maas-api-xks-ca\") pod \"b85e0598-ed75-414b-a90a-667a751cd95a\" (UID: \"b85e0598-ed75-414b-a90a-667a751cd95a\") " Oct 30 18:28:24.171069 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.171038 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b85e0598-ed75-414b-a90a-667a751cd95a-maas-api-openshift-ca" pod "b85e0598-ed75-414b-a90a-667a751cd95a" (UID: "b85e0598-ed75-414b-a90a-667a751cd95a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:24.171373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.171354 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b85e0598-ed75-414b-a90a-667a751cd95a-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:24.173671 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.173642 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b85e0598-ed75-414b-a90a-667a751cd95a-maas-api-xks-ca" pod "b85e0598-ed75-414b-a90a-667a751cd95a" (UID: "b85e0598-ed75-414b-a90a-667a751cd95a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:24.173776 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.173760 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b85e0598-ed75-414b-a90a-667a751cd95a-kube-api-access-4n6h8" pod "b85e0598-ed75-414b-a90a-667a751cd95a" (UID: "b85e0598-ed75-414b-a90a-667a751cd95a"). InnerVolumeSpecName "kube-api-access-4n6h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:24.272855 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.272820 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b85e0598-ed75-414b-a90a-667a751cd95a-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:24.272855 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.272848 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4n6h8\" (UniqueName: \"kubernetes.io/projected/b85e0598-ed75-414b-a90a-667a751cd95a-kube-api-access-4n6h8\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:24.812576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.812482 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" path="/var/lib/kubelet/pods/f7f7ebe4-e24c-4cd7-aa13-978eb635e01e/volumes" Oct 30 18:28:24.844887 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:28:24.844857 5232 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85e0598_ed75_414b_a90a_667a751cd95a.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb85e0598_ed75_414b_a90a_667a751cd95a.slice/crio-f41c820d6122212383d252363d0913e87c879ae5b7ae0fae5ab6c8fd68de7b6b\": RecentStats: unable to find data in memory cache]" Oct 30 18:28:24.854357 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.854333 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:28:24.894558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.892922 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-envoy\") pod \"37acc5cd-e689-4f1f-bb72-38794eb5e205\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " Oct 30 18:28:24.894558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.893022 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-workload-socket\") pod \"37acc5cd-e689-4f1f-bb72-38794eb5e205\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " Oct 30 18:28:24.894558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.893059 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/37acc5cd-e689-4f1f-bb72-38794eb5e205-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37acc5cd-e689-4f1f-bb72-38794eb5e205-istiod-ca-cert\") pod \"37acc5cd-e689-4f1f-bb72-38794eb5e205\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " Oct 30 18:28:24.894558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.893128 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-data\") pod \"37acc5cd-e689-4f1f-bb72-38794eb5e205\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " Oct 30 18:28:24.894558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.893168 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-token\" (UniqueName: \"kubernetes.io/projected/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-token\") pod \"37acc5cd-e689-4f1f-bb72-38794eb5e205\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " Oct 30 18:28:24.894558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.893224 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-credential-socket\") pod \"37acc5cd-e689-4f1f-bb72-38794eb5e205\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " Oct 30 18:28:24.894558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.893306 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-workload-certs\") pod \"37acc5cd-e689-4f1f-bb72-38794eb5e205\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " Oct 30 18:28:24.894558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.893366 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-podinfo\") pod \"37acc5cd-e689-4f1f-bb72-38794eb5e205\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " Oct 30 18:28:24.894558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.893430 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/37acc5cd-e689-4f1f-bb72-38794eb5e205-kube-api-access-7trt4\" (UniqueName: \"kubernetes.io/projected/37acc5cd-e689-4f1f-bb72-38794eb5e205-kube-api-access-7trt4\") pod \"37acc5cd-e689-4f1f-bb72-38794eb5e205\" (UID: \"37acc5cd-e689-4f1f-bb72-38794eb5e205\") " Oct 30 18:28:24.895860 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.895478 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-workload-socket" pod "37acc5cd-e689-4f1f-bb72-38794eb5e205" (UID: "37acc5cd-e689-4f1f-bb72-38794eb5e205"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:24.895860 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.895554 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-data" pod "37acc5cd-e689-4f1f-bb72-38794eb5e205" (UID: "37acc5cd-e689-4f1f-bb72-38794eb5e205"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:24.896057 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.895935 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/37acc5cd-e689-4f1f-bb72-38794eb5e205-istiod-ca-cert" pod "37acc5cd-e689-4f1f-bb72-38794eb5e205" (UID: "37acc5cd-e689-4f1f-bb72-38794eb5e205"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:24.896314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.896287 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-credential-socket" pod "37acc5cd-e689-4f1f-bb72-38794eb5e205" (UID: "37acc5cd-e689-4f1f-bb72-38794eb5e205"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:24.896494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.896475 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-workload-certs" pod "37acc5cd-e689-4f1f-bb72-38794eb5e205" (UID: "37acc5cd-e689-4f1f-bb72-38794eb5e205"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:24.908122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.903674 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-envoy" pod "37acc5cd-e689-4f1f-bb72-38794eb5e205" (UID: "37acc5cd-e689-4f1f-bb72-38794eb5e205"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:24.908122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.903710 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-token" pod "37acc5cd-e689-4f1f-bb72-38794eb5e205" (UID: "37acc5cd-e689-4f1f-bb72-38794eb5e205"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:24.908122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.903755 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-podinfo" pod "37acc5cd-e689-4f1f-bb72-38794eb5e205" (UID: "37acc5cd-e689-4f1f-bb72-38794eb5e205"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:24.908122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.903780 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/37acc5cd-e689-4f1f-bb72-38794eb5e205-kube-api-access-7trt4" pod "37acc5cd-e689-4f1f-bb72-38794eb5e205" (UID: "37acc5cd-e689-4f1f-bb72-38794eb5e205"). InnerVolumeSpecName "kube-api-access-7trt4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:24.942396 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.942275 5232 generic.go:356] "Generic (PLEG): container finished" podID="703d9be7-1970-4a72-8a98-7c15fa6108f1" containerID="4b60ebcd19d90f90816430447fb279f0960a6fed926fc8d2af989dc1cbee3672" exitCode=0 Oct 30 18:28:24.942396 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.942347 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" event={"ID":"703d9be7-1970-4a72-8a98-7c15fa6108f1","Type":"ContainerDied","Data":"4b60ebcd19d90f90816430447fb279f0960a6fed926fc8d2af989dc1cbee3672"} Oct 30 18:28:24.944956 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.944925 5232 generic.go:356] "Generic (PLEG): container finished" podID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerID="fbe4c68188085bdea85053a1d780b92618cad40a8eccaf2bc2fda31b377c5341" exitCode=0 Oct 30 18:28:24.945102 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.945029 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" Oct 30 18:28:24.945102 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.945082 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" event={"ID":"37acc5cd-e689-4f1f-bb72-38794eb5e205","Type":"ContainerDied","Data":"fbe4c68188085bdea85053a1d780b92618cad40a8eccaf2bc2fda31b377c5341"} Oct 30 18:28:24.945216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.945117 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw" event={"ID":"37acc5cd-e689-4f1f-bb72-38794eb5e205","Type":"ContainerDied","Data":"806ddbade2a1c1b63ede905361c73c8f27f54f3c7fa0bf537c89c0fc011a85fb"} Oct 30 18:28:24.945216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.945140 5232 scope.go:122] "RemoveContainer" containerID="fbe4c68188085bdea85053a1d780b92618cad40a8eccaf2bc2fda31b377c5341" Oct 30 18:28:24.958986 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.957933 5232 generic.go:356] "Generic (PLEG): container finished" podID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerID="f2767476296f1939812236ce78d03b4f0677055fa6bdde7665176edac0a7a606" exitCode=0 Oct 30 18:28:24.958986 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.958024 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" event={"ID":"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998","Type":"ContainerDied","Data":"f2767476296f1939812236ce78d03b4f0677055fa6bdde7665176edac0a7a606"} Oct 30 18:28:24.962528 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.961945 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx" Oct 30 18:28:24.962528 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.962073 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx" event={"ID":"b85e0598-ed75-414b-a90a-667a751cd95a","Type":"ContainerDied","Data":"f41c820d6122212383d252363d0913e87c879ae5b7ae0fae5ab6c8fd68de7b6b"} Oct 30 18:28:24.962528 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.962116 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f41c820d6122212383d252363d0913e87c879ae5b7ae0fae5ab6c8fd68de7b6b" Oct 30 18:28:24.987582 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.987556 5232 scope.go:122] "RemoveContainer" containerID="0e97b95f73eecec35274b0726fc5500f57a6051ebfa5dc281f87edf566737e47" Oct 30 18:28:24.994927 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.994903 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:28:24.995580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.995551 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:24.995709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.995586 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:24.995709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.995601 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:24.995709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.995615 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:24.995709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.995627 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7trt4\" (UniqueName: \"kubernetes.io/projected/37acc5cd-e689-4f1f-bb72-38794eb5e205-kube-api-access-7trt4\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:24.995709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.995642 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:24.995709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.995655 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:24.995709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.995667 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/37acc5cd-e689-4f1f-bb72-38794eb5e205-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:24.995709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.995679 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/37acc5cd-e689-4f1f-bb72-38794eb5e205-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:24.997889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:24.997815 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:25.003785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.003744 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw"] Oct 30 18:28:25.006181 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.006141 5232 scope.go:122] "RemoveContainer" containerID="fbe4c68188085bdea85053a1d780b92618cad40a8eccaf2bc2fda31b377c5341" Oct 30 18:28:25.013603 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:28:25.006817 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fbe4c68188085bdea85053a1d780b92618cad40a8eccaf2bc2fda31b377c5341\": container with ID starting with fbe4c68188085bdea85053a1d780b92618cad40a8eccaf2bc2fda31b377c5341 not found: ID does not exist" containerID="fbe4c68188085bdea85053a1d780b92618cad40a8eccaf2bc2fda31b377c5341" Oct 30 18:28:25.013603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.006855 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fbe4c68188085bdea85053a1d780b92618cad40a8eccaf2bc2fda31b377c5341"} err="failed to get container status \"fbe4c68188085bdea85053a1d780b92618cad40a8eccaf2bc2fda31b377c5341\": rpc error: code = NotFound desc = could not find container \"fbe4c68188085bdea85053a1d780b92618cad40a8eccaf2bc2fda31b377c5341\": container with ID starting with fbe4c68188085bdea85053a1d780b92618cad40a8eccaf2bc2fda31b377c5341 not found: ID does not exist" Oct 30 18:28:25.013603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.006880 5232 scope.go:122] "RemoveContainer" containerID="0e97b95f73eecec35274b0726fc5500f57a6051ebfa5dc281f87edf566737e47" Oct 30 18:28:25.013603 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:28:25.007465 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0e97b95f73eecec35274b0726fc5500f57a6051ebfa5dc281f87edf566737e47\": container with ID starting with 0e97b95f73eecec35274b0726fc5500f57a6051ebfa5dc281f87edf566737e47 not found: ID does not exist" containerID="0e97b95f73eecec35274b0726fc5500f57a6051ebfa5dc281f87edf566737e47" Oct 30 18:28:25.013603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.007520 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0e97b95f73eecec35274b0726fc5500f57a6051ebfa5dc281f87edf566737e47"} err="failed to get container status \"0e97b95f73eecec35274b0726fc5500f57a6051ebfa5dc281f87edf566737e47\": rpc error: code = NotFound desc = could not find container \"0e97b95f73eecec35274b0726fc5500f57a6051ebfa5dc281f87edf566737e47\": container with ID starting with 0e97b95f73eecec35274b0726fc5500f57a6051ebfa5dc281f87edf566737e47 not found: ID does not exist" Oct 30 18:28:25.013603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.007546 5232 scope.go:122] "RemoveContainer" containerID="c20ab9c1c9576eb5e86497bb8ec8e3ccd1787a00cfb6798231e7d1bb123f8455" Oct 30 18:28:25.013603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.011948 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-mggdw"] Oct 30 18:28:25.096430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.096384 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-credential-socket\") pod \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " Oct 30 18:28:25.096616 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.096478 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/703d9be7-1970-4a72-8a98-7c15fa6108f1-kube-api-access-sk7rc\" (UniqueName: \"kubernetes.io/projected/703d9be7-1970-4a72-8a98-7c15fa6108f1-kube-api-access-sk7rc\") pod \"703d9be7-1970-4a72-8a98-7c15fa6108f1\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " Oct 30 18:28:25.096616 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.096529 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/703d9be7-1970-4a72-8a98-7c15fa6108f1-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/703d9be7-1970-4a72-8a98-7c15fa6108f1-istiod-ca-cert\") pod \"703d9be7-1970-4a72-8a98-7c15fa6108f1\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " Oct 30 18:28:25.096616 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.096584 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-kube-api-access-xrk72\" (UniqueName: \"kubernetes.io/projected/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-kube-api-access-xrk72\") pod \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " Oct 30 18:28:25.096784 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.096639 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-credential-socket\") pod \"703d9be7-1970-4a72-8a98-7c15fa6108f1\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " Oct 30 18:28:25.097001 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.096975 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/703d9be7-1970-4a72-8a98-7c15fa6108f1-istiod-ca-cert" pod "703d9be7-1970-4a72-8a98-7c15fa6108f1" (UID: "703d9be7-1970-4a72-8a98-7c15fa6108f1"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:25.097158 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.097136 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-workload-socket\") pod \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " Oct 30 18:28:25.097292 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.097272 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-credential-socket" pod "703d9be7-1970-4a72-8a98-7c15fa6108f1" (UID: "703d9be7-1970-4a72-8a98-7c15fa6108f1"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.097365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.097281 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-workload-socket\") pod \"703d9be7-1970-4a72-8a98-7c15fa6108f1\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " Oct 30 18:28:25.097426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.097391 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-token\" (UniqueName: \"kubernetes.io/projected/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-token\") pod \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " Oct 30 18:28:25.097485 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.097467 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-workload-socket" pod "703d9be7-1970-4a72-8a98-7c15fa6108f1" (UID: "703d9be7-1970-4a72-8a98-7c15fa6108f1"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.097485 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.097479 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-data\") pod \"703d9be7-1970-4a72-8a98-7c15fa6108f1\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " Oct 30 18:28:25.097642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.097552 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-podinfo\") pod \"703d9be7-1970-4a72-8a98-7c15fa6108f1\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " Oct 30 18:28:25.097642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.097587 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-podinfo\") pod \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " Oct 30 18:28:25.097642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.097625 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-envoy\") pod \"703d9be7-1970-4a72-8a98-7c15fa6108f1\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " Oct 30 18:28:25.097750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.097658 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-workload-socket" pod "4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" (UID: "4f5b3876-dc3b-4f39-a6f1-4591ce4a9998"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.097750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.097689 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-workload-certs\") pod \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " Oct 30 18:28:25.097750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.097720 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istiod-ca-cert\") pod \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " Oct 30 18:28:25.097851 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.097766 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-envoy\") pod \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " Oct 30 18:28:25.097851 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.097810 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-data\") pod \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\" (UID: \"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998\") " Oct 30 18:28:25.097851 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.097823 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-data" pod "703d9be7-1970-4a72-8a98-7c15fa6108f1" (UID: "703d9be7-1970-4a72-8a98-7c15fa6108f1"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.097851 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.097837 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-token\" (UniqueName: \"kubernetes.io/projected/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-token\") pod \"703d9be7-1970-4a72-8a98-7c15fa6108f1\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " Oct 30 18:28:25.097977 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.097878 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-workload-certs\") pod \"703d9be7-1970-4a72-8a98-7c15fa6108f1\" (UID: \"703d9be7-1970-4a72-8a98-7c15fa6108f1\") " Oct 30 18:28:25.098550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.098522 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-workload-certs" pod "4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" (UID: "4f5b3876-dc3b-4f39-a6f1-4591ce4a9998"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.098656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.098578 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/703d9be7-1970-4a72-8a98-7c15fa6108f1-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.098656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.098599 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.098656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.098613 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.098656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.098627 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.098656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.098640 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.098943 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.098836 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-data" pod "4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" (UID: "4f5b3876-dc3b-4f39-a6f1-4591ce4a9998"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.099003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.098966 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istiod-ca-cert" pod "4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" (UID: "4f5b3876-dc3b-4f39-a6f1-4591ce4a9998"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:25.099145 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.099121 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-credential-socket" pod "4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" (UID: "4f5b3876-dc3b-4f39-a6f1-4591ce4a9998"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.102932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.102903 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-workload-certs" pod "703d9be7-1970-4a72-8a98-7c15fa6108f1" (UID: "703d9be7-1970-4a72-8a98-7c15fa6108f1"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.107333 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.104904 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-podinfo" pod "703d9be7-1970-4a72-8a98-7c15fa6108f1" (UID: "703d9be7-1970-4a72-8a98-7c15fa6108f1"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:25.107333 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.104963 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-kube-api-access-xrk72" pod "4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" (UID: "4f5b3876-dc3b-4f39-a6f1-4591ce4a9998"). InnerVolumeSpecName "kube-api-access-xrk72". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:25.107333 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.105016 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/703d9be7-1970-4a72-8a98-7c15fa6108f1-kube-api-access-sk7rc" pod "703d9be7-1970-4a72-8a98-7c15fa6108f1" (UID: "703d9be7-1970-4a72-8a98-7c15fa6108f1"). InnerVolumeSpecName "kube-api-access-sk7rc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:25.107333 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.106036 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-podinfo" pod "4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" (UID: "4f5b3876-dc3b-4f39-a6f1-4591ce4a9998"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:25.107333 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.106152 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-token" pod "703d9be7-1970-4a72-8a98-7c15fa6108f1" (UID: "703d9be7-1970-4a72-8a98-7c15fa6108f1"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:25.107333 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.107035 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-envoy" pod "4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" (UID: "4f5b3876-dc3b-4f39-a6f1-4591ce4a9998"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.107764 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.107362 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-envoy" pod "703d9be7-1970-4a72-8a98-7c15fa6108f1" (UID: "703d9be7-1970-4a72-8a98-7c15fa6108f1"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:25.108964 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.108932 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-token" pod "4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" (UID: "4f5b3876-dc3b-4f39-a6f1-4591ce4a9998"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:25.200538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.199874 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sk7rc\" (UniqueName: \"kubernetes.io/projected/703d9be7-1970-4a72-8a98-7c15fa6108f1-kube-api-access-sk7rc\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.200538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.199911 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xrk72\" (UniqueName: \"kubernetes.io/projected/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-kube-api-access-xrk72\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.200538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.199926 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.200538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.199939 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.200538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.199951 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.200538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.199963 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.200538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.199977 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.200538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.199989 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.200538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.200000 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.200538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.200013 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.200538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.200024 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/703d9be7-1970-4a72-8a98-7c15fa6108f1-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.200538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.200036 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/703d9be7-1970-4a72-8a98-7c15fa6108f1-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.200538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.200051 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:25.793856 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.793820 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm"] Oct 30 18:28:25.794475 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794461 5232 cpu_manager.go:414] "containerMap: removing container" podUID="703d9be7-1970-4a72-8a98-7c15fa6108f1" containerName="istio-proxy" Oct 30 18:28:25.794558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794477 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="703d9be7-1970-4a72-8a98-7c15fa6108f1" containerName="istio-proxy" Oct 30 18:28:25.794558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794493 5232 cpu_manager.go:414] "containerMap: removing container" podUID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerName="istio-proxy" Oct 30 18:28:25.794558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794499 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerName="istio-proxy" Oct 30 18:28:25.794558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794543 5232 cpu_manager.go:414] "containerMap: removing container" podUID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerName="istio-proxy" Oct 30 18:28:25.794558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794551 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerName="istio-proxy" Oct 30 18:28:25.794712 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794565 5232 cpu_manager.go:414] "containerMap: removing container" podUID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerName="istio-proxy" Oct 30 18:28:25.794712 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794571 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerName="istio-proxy" Oct 30 18:28:25.794712 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794593 5232 cpu_manager.go:414] "containerMap: removing container" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" Oct 30 18:28:25.794712 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794598 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" Oct 30 18:28:25.794712 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794606 5232 cpu_manager.go:414] "containerMap: removing container" podUID="b85e0598-ed75-414b-a90a-667a751cd95a" containerName="revoke-keys" Oct 30 18:28:25.794712 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794610 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b85e0598-ed75-414b-a90a-667a751cd95a" containerName="revoke-keys" Oct 30 18:28:25.794712 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794616 5232 cpu_manager.go:414] "containerMap: removing container" podUID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerName="istio-proxy" Oct 30 18:28:25.794712 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794620 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerName="istio-proxy" Oct 30 18:28:25.794950 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794732 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerName="istio-proxy" Oct 30 18:28:25.794950 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794742 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerName="istio-proxy" Oct 30 18:28:25.794950 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794750 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" containerName="istio-proxy" Oct 30 18:28:25.794950 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794757 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="f7f7ebe4-e24c-4cd7-aa13-978eb635e01e" containerName="main" Oct 30 18:28:25.794950 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794763 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="b85e0598-ed75-414b-a90a-667a751cd95a" containerName="revoke-keys" Oct 30 18:28:25.794950 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794770 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="37acc5cd-e689-4f1f-bb72-38794eb5e205" containerName="istio-proxy" Oct 30 18:28:25.794950 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.794778 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="703d9be7-1970-4a72-8a98-7c15fa6108f1" containerName="istio-proxy" Oct 30 18:28:25.800139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.800117 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.807328 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.807300 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-workload-socket\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.807480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.807353 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de317989-41ca-479f-bcc1-9c7b5a166e70-istiod-ca-cert\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.807627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.807601 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-podinfo\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.807713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.807694 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-token\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.807782 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.807741 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-credential-socket\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.807842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.807798 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8g6j\" (UniqueName: \"kubernetes.io/projected/de317989-41ca-479f-bcc1-9c7b5a166e70-kube-api-access-v8g6j\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.808066 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.807865 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-envoy\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.808120 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.808088 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-workload-certs\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.808174 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.808156 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-data\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.815448 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.815413 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-1b0848-openshift-default-dockercfg-cw6rr\"" Oct 30 18:28:25.831451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.831415 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm"] Oct 30 18:28:25.910166 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.910133 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-podinfo\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.910350 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.910188 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-token\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.910350 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.910205 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-credential-socket\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.910637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.910612 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-credential-socket\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.910785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.910677 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8g6j\" (UniqueName: \"kubernetes.io/projected/de317989-41ca-479f-bcc1-9c7b5a166e70-kube-api-access-v8g6j\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.910785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.910737 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-envoy\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.910785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.910777 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-workload-certs\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.910979 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.910839 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-data\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.910979 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.910931 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-workload-socket\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.911096 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.910978 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de317989-41ca-479f-bcc1-9c7b5a166e70-istiod-ca-cert\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.911156 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.911126 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-workload-certs\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.911320 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.911299 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-data\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.911403 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.911381 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-workload-socket\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.911711 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.911694 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de317989-41ca-479f-bcc1-9c7b5a166e70-istiod-ca-cert\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.912929 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.912912 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-podinfo\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.913048 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.913031 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-envoy\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.930584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.930541 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-token\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:25.930689 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:25.930617 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8g6j\" (UniqueName: \"kubernetes.io/projected/de317989-41ca-479f-bcc1-9c7b5a166e70-kube-api-access-v8g6j\") pod \"e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:26.038560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.034094 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" event={"ID":"4f5b3876-dc3b-4f39-a6f1-4591ce4a9998","Type":"ContainerDied","Data":"62f9a3fc83a586a639e08fc877f239337a28bdc41956b644fb879e195de3fc1a"} Oct 30 18:28:26.038560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.034189 5232 scope.go:122] "RemoveContainer" containerID="f2767476296f1939812236ce78d03b4f0677055fa6bdde7665176edac0a7a606" Oct 30 18:28:26.038560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.034396 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24" Oct 30 18:28:26.112480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.109915 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" event={"ID":"703d9be7-1970-4a72-8a98-7c15fa6108f1","Type":"ContainerDied","Data":"fcea701bb3c0066f2e862841c21ba3c8163d8b65a8246737c245e41d6dab1e24"} Oct 30 18:28:26.112480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.110102 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w" Oct 30 18:28:26.135752 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.133783 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:26.196970 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.196573 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24"] Oct 30 18:28:26.203062 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.203027 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-rrv24"] Oct 30 18:28:26.203740 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.203721 5232 scope.go:122] "RemoveContainer" containerID="4b60ebcd19d90f90816430447fb279f0960a6fed926fc8d2af989dc1cbee3672" Oct 30 18:28:26.241622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.241593 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w"] Oct 30 18:28:26.248272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.248240 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-ae3373-openshift-default-78486566c-sbz9w"] Oct 30 18:28:26.482464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.482434 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm"] Oct 30 18:28:26.482991 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:28:26.482969 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde317989_41ca_479f_bcc1_9c7b5a166e70.slice/crio-54f4b77719cf2516a24cd3cd6da4af0ea44d4b03b88df847a414dae560aa52f5 WatchSource:0}: Error finding container 54f4b77719cf2516a24cd3cd6da4af0ea44d4b03b88df847a414dae560aa52f5: Status 404 returned error can't find the container with id 54f4b77719cf2516a24cd3cd6da4af0ea44d4b03b88df847a414dae560aa52f5 Oct 30 18:28:26.486292 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.486260 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:28:26.486358 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.486330 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:28:26.486392 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.486360 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:28:26.805152 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.805120 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="37acc5cd-e689-4f1f-bb72-38794eb5e205" path="/var/lib/kubelet/pods/37acc5cd-e689-4f1f-bb72-38794eb5e205/volumes" Oct 30 18:28:26.805874 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.805831 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f5b3876-dc3b-4f39-a6f1-4591ce4a9998" path="/var/lib/kubelet/pods/4f5b3876-dc3b-4f39-a6f1-4591ce4a9998/volumes" Oct 30 18:28:26.806363 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.806344 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="703d9be7-1970-4a72-8a98-7c15fa6108f1" path="/var/lib/kubelet/pods/703d9be7-1970-4a72-8a98-7c15fa6108f1/volumes" Oct 30 18:28:26.928900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.928870 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw"] Oct 30 18:28:26.934373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.934353 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" Oct 30 18:28:26.949727 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.949693 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-b-69c44c-serving-cert\"" Oct 30 18:28:26.952589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:26.952565 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw"] Oct 30 18:28:27.036621 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:28:27.036587 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:28:27.073380 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.073291 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cfb4330-a715-4d7b-aa34-560dfbecf89c-maas-api-tls\") pod \"maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw\" (UID: \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" Oct 30 18:28:27.073602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.073435 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wwzwm\" (UniqueName: \"kubernetes.io/projected/8cfb4330-a715-4d7b-aa34-560dfbecf89c-kube-api-access-wwzwm\") pod \"maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw\" (UID: \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" Oct 30 18:28:27.073602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.073465 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfb4330-a715-4d7b-aa34-560dfbecf89c-metrics-certs\") pod \"maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw\" (UID: \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" Oct 30 18:28:27.073737 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.073599 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cfb4330-a715-4d7b-aa34-560dfbecf89c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw\" (UID: \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" Oct 30 18:28:27.183872 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.183672 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wwzwm\" (UniqueName: \"kubernetes.io/projected/8cfb4330-a715-4d7b-aa34-560dfbecf89c-kube-api-access-wwzwm\") pod \"maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw\" (UID: \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" Oct 30 18:28:27.183872 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.183742 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfb4330-a715-4d7b-aa34-560dfbecf89c-metrics-certs\") pod \"maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw\" (UID: \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" Oct 30 18:28:27.183872 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.183782 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cfb4330-a715-4d7b-aa34-560dfbecf89c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw\" (UID: \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" Oct 30 18:28:27.184627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.184029 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cfb4330-a715-4d7b-aa34-560dfbecf89c-maas-api-tls\") pod \"maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw\" (UID: \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" Oct 30 18:28:27.187711 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.187642 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cfb4330-a715-4d7b-aa34-560dfbecf89c-openshift-service-ca\") pod \"maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw\" (UID: \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" Oct 30 18:28:27.226230 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.226185 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wwzwm\" (UniqueName: \"kubernetes.io/projected/8cfb4330-a715-4d7b-aa34-560dfbecf89c-kube-api-access-wwzwm\") pod \"maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw\" (UID: \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" Oct 30 18:28:27.226230 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.226214 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cfb4330-a715-4d7b-aa34-560dfbecf89c-maas-api-tls\") pod \"maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw\" (UID: \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" Oct 30 18:28:27.233400 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.233362 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfb4330-a715-4d7b-aa34-560dfbecf89c-metrics-certs\") pod \"maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw\" (UID: \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" Oct 30 18:28:27.242860 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.240881 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" event={"ID":"de317989-41ca-479f-bcc1-9c7b5a166e70","Type":"ContainerStarted","Data":"23575259bdca43d9ceb0b2822da983b9abd29ebdf649da0ce863a0f7385a4b6a"} Oct 30 18:28:27.242860 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.240959 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" event={"ID":"de317989-41ca-479f-bcc1-9c7b5a166e70","Type":"ContainerStarted","Data":"54f4b77719cf2516a24cd3cd6da4af0ea44d4b03b88df847a414dae560aa52f5"} Oct 30 18:28:27.252063 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.251816 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" Oct 30 18:28:27.282733 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.282686 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" podStartSLOduration=2.282670118 podStartE2EDuration="2.282670118s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:27.282337875 +0000 UTC m=+1606.598758196" watchObservedRunningTime="2026-10-30 18:28:27.282670118 +0000 UTC m=+1606.599090441" Oct 30 18:28:27.310415 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.310376 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689"] Oct 30 18:28:27.316333 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.316311 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689" Oct 30 18:28:27.341161 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.339490 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689"] Oct 30 18:28:27.352572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.347234 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:28:27.358577 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.347489 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:28:27.390206 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.389045 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qlsc8\" (UniqueName: \"kubernetes.io/projected/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-kube-api-access-qlsc8\") pod \"maas-api-revoke-sub-4fe1ddc9-hd689\" (UID: \"0cdcd6ec-0417-4dd7-8ca7-032c12b859e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689" Oct 30 18:28:27.390206 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.389136 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fe1ddc9-hd689\" (UID: \"0cdcd6ec-0417-4dd7-8ca7-032c12b859e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689" Oct 30 18:28:27.390206 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.389172 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fe1ddc9-hd689\" (UID: \"0cdcd6ec-0417-4dd7-8ca7-032c12b859e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689" Oct 30 18:28:27.491129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.491096 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qlsc8\" (UniqueName: \"kubernetes.io/projected/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-kube-api-access-qlsc8\") pod \"maas-api-revoke-sub-4fe1ddc9-hd689\" (UID: \"0cdcd6ec-0417-4dd7-8ca7-032c12b859e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689" Oct 30 18:28:27.491336 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.491152 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fe1ddc9-hd689\" (UID: \"0cdcd6ec-0417-4dd7-8ca7-032c12b859e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689" Oct 30 18:28:27.491336 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.491174 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fe1ddc9-hd689\" (UID: \"0cdcd6ec-0417-4dd7-8ca7-032c12b859e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689" Oct 30 18:28:27.491859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.491834 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fe1ddc9-hd689\" (UID: \"0cdcd6ec-0417-4dd7-8ca7-032c12b859e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689" Oct 30 18:28:27.493713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.493691 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fe1ddc9-hd689\" (UID: \"0cdcd6ec-0417-4dd7-8ca7-032c12b859e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689" Oct 30 18:28:27.512630 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.512601 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qlsc8\" (UniqueName: \"kubernetes.io/projected/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-kube-api-access-qlsc8\") pod \"maas-api-revoke-sub-4fe1ddc9-hd689\" (UID: \"0cdcd6ec-0417-4dd7-8ca7-032c12b859e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689" Oct 30 18:28:27.565104 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.565076 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw"] Oct 30 18:28:27.565602 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:28:27.565578 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cfb4330_a715_4d7b_aa34_560dfbecf89c.slice/crio-f994465b2f1e60397ab159b63cd874595b94b8b01fdcfd3ef1e8af067e124849 WatchSource:0}: Error finding container f994465b2f1e60397ab159b63cd874595b94b8b01fdcfd3ef1e8af067e124849: Status 404 returned error can't find the container with id f994465b2f1e60397ab159b63cd874595b94b8b01fdcfd3ef1e8af067e124849 Oct 30 18:28:27.567080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.567063 5232 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:28:27.661741 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.661659 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689" Oct 30 18:28:27.970467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:27.970442 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689"] Oct 30 18:28:27.970611 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:28:27.970444 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0cdcd6ec_0417_4dd7_8ca7_032c12b859e3.slice/crio-99ca39e93ac3698d09f7b7ffc396e4c41e483549761089265441e51ad9542391 WatchSource:0}: Error finding container 99ca39e93ac3698d09f7b7ffc396e4c41e483549761089265441e51ad9542391: Status 404 returned error can't find the container with id 99ca39e93ac3698d09f7b7ffc396e4c41e483549761089265441e51ad9542391 Oct 30 18:28:28.138578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:28.138546 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:28.144863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:28.144833 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:28.254341 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:28.254293 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689" event={"ID":"0cdcd6ec-0417-4dd7-8ca7-032c12b859e3","Type":"ContainerStarted","Data":"99ca39e93ac3698d09f7b7ffc396e4c41e483549761089265441e51ad9542391"} Oct 30 18:28:28.256046 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:28.256016 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" event={"ID":"8cfb4330-a715-4d7b-aa34-560dfbecf89c","Type":"ContainerStarted","Data":"85a10794fe20e128a1f4d3a833bd0adee7c81760f5bdfa32c875eaf6206409f6"} Oct 30 18:28:28.256184 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:28.256068 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" event={"ID":"8cfb4330-a715-4d7b-aa34-560dfbecf89c","Type":"ContainerStarted","Data":"f994465b2f1e60397ab159b63cd874595b94b8b01fdcfd3ef1e8af067e124849"} Oct 30 18:28:28.256377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:28.256351 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:28.256546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:28.256445 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" Oct 30 18:28:28.257273 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:28.257257 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:28:28.297811 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:28.297766 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" podStartSLOduration=1.718187057 podStartE2EDuration="2.297754421s" totalImagesPullingTime="579.567364ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:28.295279818 +0000 UTC m=+1607.611700157" watchObservedRunningTime="2026-10-30 18:28:28.297754421 +0000 UTC m=+1607.614174737" Oct 30 18:28:29.000850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:29.000818 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" podUID="771db49d-214d-4901-8885-50e40d3d4d88" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:28:29.001018 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:29.000817 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" podUID="552542b1-fdd2-417f-88a5-614f087e6228" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:28:29.265047 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:29.264954 5232 generic.go:356] "Generic (PLEG): container finished" podID="0cdcd6ec-0417-4dd7-8ca7-032c12b859e3" containerID="923e86a84ff85061a5bce8061ef77980795d144addff6a42a903a609e0c8bb38" exitCode=0 Oct 30 18:28:29.265469 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:29.265050 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689" event={"ID":"0cdcd6ec-0417-4dd7-8ca7-032c12b859e3","Type":"ContainerDied","Data":"923e86a84ff85061a5bce8061ef77980795d144addff6a42a903a609e0c8bb38"} Oct 30 18:28:30.435941 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:30.435913 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689" Oct 30 18:28:30.583615 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:30.583532 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b"] Oct 30 18:28:30.632222 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:30.632178 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-kube-api-access-qlsc8\" (UniqueName: \"kubernetes.io/projected/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-kube-api-access-qlsc8\") pod \"0cdcd6ec-0417-4dd7-8ca7-032c12b859e3\" (UID: \"0cdcd6ec-0417-4dd7-8ca7-032c12b859e3\") " Oct 30 18:28:30.632719 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:30.632693 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-maas-api-xks-ca\") pod \"0cdcd6ec-0417-4dd7-8ca7-032c12b859e3\" (UID: \"0cdcd6ec-0417-4dd7-8ca7-032c12b859e3\") " Oct 30 18:28:30.633042 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:30.633023 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-maas-api-openshift-ca\") pod \"0cdcd6ec-0417-4dd7-8ca7-032c12b859e3\" (UID: \"0cdcd6ec-0417-4dd7-8ca7-032c12b859e3\") " Oct 30 18:28:30.633634 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:30.633525 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-maas-api-openshift-ca" pod "0cdcd6ec-0417-4dd7-8ca7-032c12b859e3" (UID: "0cdcd6ec-0417-4dd7-8ca7-032c12b859e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:30.634454 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:30.634428 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:30.638004 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:30.637589 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-kube-api-access-qlsc8" pod "0cdcd6ec-0417-4dd7-8ca7-032c12b859e3" (UID: "0cdcd6ec-0417-4dd7-8ca7-032c12b859e3"). InnerVolumeSpecName "kube-api-access-qlsc8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:30.642665 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:30.642626 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-maas-api-xks-ca" pod "0cdcd6ec-0417-4dd7-8ca7-032c12b859e3" (UID: "0cdcd6ec-0417-4dd7-8ca7-032c12b859e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:30.735425 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:30.735390 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qlsc8\" (UniqueName: \"kubernetes.io/projected/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-kube-api-access-qlsc8\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:30.735425 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:30.735421 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:31.326486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:31.326439 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689" event={"ID":"0cdcd6ec-0417-4dd7-8ca7-032c12b859e3","Type":"ContainerDied","Data":"99ca39e93ac3698d09f7b7ffc396e4c41e483549761089265441e51ad9542391"} Oct 30 18:28:31.326704 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:31.326521 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99ca39e93ac3698d09f7b7ffc396e4c41e483549761089265441e51ad9542391" Oct 30 18:28:31.326704 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:31.326593 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689" Oct 30 18:28:32.375738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:32.368610 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k"] Oct 30 18:28:32.375738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:32.369703 5232 cpu_manager.go:414] "containerMap: removing container" podUID="0cdcd6ec-0417-4dd7-8ca7-032c12b859e3" containerName="revoke-keys" Oct 30 18:28:32.375738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:32.369723 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0cdcd6ec-0417-4dd7-8ca7-032c12b859e3" containerName="revoke-keys" Oct 30 18:28:32.375738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:32.372986 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="0cdcd6ec-0417-4dd7-8ca7-032c12b859e3" containerName="revoke-keys" Oct 30 18:28:32.381939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:32.381910 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k"] Oct 30 18:28:32.382130 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:32.382031 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k" Oct 30 18:28:32.397034 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:32.397010 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-1b0848\"" Oct 30 18:28:32.409186 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:32.409142 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-1b0848-dockercfg-gllp7\"" Oct 30 18:28:32.483311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:32.483277 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9721f2-b5b3-4ebb-8677-3e5f1192fe74-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k\" (UID: \"0f9721f2-b5b3-4ebb-8677-3e5f1192fe74\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k" Oct 30 18:28:32.483483 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:32.483325 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vszj9\" (UniqueName: \"kubernetes.io/projected/0f9721f2-b5b3-4ebb-8677-3e5f1192fe74-kube-api-access-vszj9\") pod \"payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k\" (UID: \"0f9721f2-b5b3-4ebb-8677-3e5f1192fe74\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k" Oct 30 18:28:32.601534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:32.596916 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9721f2-b5b3-4ebb-8677-3e5f1192fe74-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k\" (UID: \"0f9721f2-b5b3-4ebb-8677-3e5f1192fe74\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k" Oct 30 18:28:32.601534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:32.596982 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vszj9\" (UniqueName: \"kubernetes.io/projected/0f9721f2-b5b3-4ebb-8677-3e5f1192fe74-kube-api-access-vszj9\") pod \"payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k\" (UID: \"0f9721f2-b5b3-4ebb-8677-3e5f1192fe74\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k" Oct 30 18:28:32.601534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:32.598467 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9721f2-b5b3-4ebb-8677-3e5f1192fe74-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k\" (UID: \"0f9721f2-b5b3-4ebb-8677-3e5f1192fe74\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k" Oct 30 18:28:32.624530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:32.623672 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vszj9\" (UniqueName: \"kubernetes.io/projected/0f9721f2-b5b3-4ebb-8677-3e5f1192fe74-kube-api-access-vszj9\") pod \"payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k\" (UID: \"0f9721f2-b5b3-4ebb-8677-3e5f1192fe74\") " pod="openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k" Oct 30 18:28:32.711313 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:32.711230 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k" Oct 30 18:28:33.012889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:33.012857 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k"] Oct 30 18:28:33.016419 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:28:33.016389 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f9721f2_b5b3_4ebb_8677_3e5f1192fe74.slice/crio-8b04e11c0febce3dd25731a5e789520ee3ccf48689b632c5f1df9891a3324c8d WatchSource:0}: Error finding container 8b04e11c0febce3dd25731a5e789520ee3ccf48689b632c5f1df9891a3324c8d: Status 404 returned error can't find the container with id 8b04e11c0febce3dd25731a5e789520ee3ccf48689b632c5f1df9891a3324c8d Oct 30 18:28:33.391154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:33.391058 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k" event={"ID":"0f9721f2-b5b3-4ebb-8677-3e5f1192fe74","Type":"ContainerStarted","Data":"cab48da4da62814999449d9cbf05c0f3e720f4fa46ca0b26e4cead20b76db908"} Oct 30 18:28:33.391154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:33.391124 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k" event={"ID":"0f9721f2-b5b3-4ebb-8677-3e5f1192fe74","Type":"ContainerStarted","Data":"8b04e11c0febce3dd25731a5e789520ee3ccf48689b632c5f1df9891a3324c8d"} Oct 30 18:28:33.391154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:33.391156 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k" Oct 30 18:28:33.433715 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:33.433658 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k" podStartSLOduration=1.433644331 podStartE2EDuration="1.433644331s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:33.431937685 +0000 UTC m=+1612.748358010" watchObservedRunningTime="2026-10-30 18:28:33.433644331 +0000 UTC m=+1612.750064685" Oct 30 18:28:34.328535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:34.327691 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" Oct 30 18:28:34.606490 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:34.606410 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx"] Oct 30 18:28:34.609701 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:34.609676 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-e5d168-bp4nx"] Oct 30 18:28:34.818073 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:34.818036 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b85e0598-ed75-414b-a90a-667a751cd95a" path="/var/lib/kubelet/pods/b85e0598-ed75-414b-a90a-667a751cd95a/volumes" Oct 30 18:28:35.584959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:35.584916 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b" podUID="9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2" containerName="payload-processing" containerID="cri-o://43c24ca11c1087509620908cd23a9805ab14bdd5e6db8139dcae25cef92df458" gracePeriod=25 Oct 30 18:28:35.862144 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:35.862109 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b" Oct 30 18:28:35.966153 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:35.966124 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2-kube-api-access-6fwsr\" (UniqueName: \"kubernetes.io/projected/9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2-kube-api-access-6fwsr\") pod \"9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2\" (UID: \"9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2\") " Oct 30 18:28:35.966336 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:35.966321 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2-plugins-config-volume\") pod \"9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2\" (UID: \"9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2\") " Oct 30 18:28:35.966780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:35.966756 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2-plugins-config-volume" pod "9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2" (UID: "9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:35.966883 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:35.966870 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:35.968377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:35.968353 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2-kube-api-access-6fwsr" pod "9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2" (UID: "9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2"). InnerVolumeSpecName "kube-api-access-6fwsr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:36.068348 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:36.068316 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fwsr\" (UniqueName: \"kubernetes.io/projected/9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2-kube-api-access-6fwsr\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:36.419670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:36.419633 5232 generic.go:356] "Generic (PLEG): container finished" podID="9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2" containerID="43c24ca11c1087509620908cd23a9805ab14bdd5e6db8139dcae25cef92df458" exitCode=0 Oct 30 18:28:36.419866 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:36.419682 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b" Oct 30 18:28:36.419866 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:36.419719 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b" event={"ID":"9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2","Type":"ContainerDied","Data":"43c24ca11c1087509620908cd23a9805ab14bdd5e6db8139dcae25cef92df458"} Oct 30 18:28:36.419866 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:36.419806 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b" event={"ID":"9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2","Type":"ContainerDied","Data":"8f5386ce5962df91ac3d128c077017bf104eb44248c43058cdb48b741905b832"} Oct 30 18:28:36.419866 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:36.419836 5232 scope.go:122] "RemoveContainer" containerID="43c24ca11c1087509620908cd23a9805ab14bdd5e6db8139dcae25cef92df458" Oct 30 18:28:36.431219 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:36.431198 5232 scope.go:122] "RemoveContainer" containerID="43c24ca11c1087509620908cd23a9805ab14bdd5e6db8139dcae25cef92df458" Oct 30 18:28:36.431570 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:28:36.431502 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43c24ca11c1087509620908cd23a9805ab14bdd5e6db8139dcae25cef92df458\": container with ID starting with 43c24ca11c1087509620908cd23a9805ab14bdd5e6db8139dcae25cef92df458 not found: ID does not exist" containerID="43c24ca11c1087509620908cd23a9805ab14bdd5e6db8139dcae25cef92df458" Oct 30 18:28:36.431658 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:36.431579 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43c24ca11c1087509620908cd23a9805ab14bdd5e6db8139dcae25cef92df458"} err="failed to get container status \"43c24ca11c1087509620908cd23a9805ab14bdd5e6db8139dcae25cef92df458\": rpc error: code = NotFound desc = could not find container \"43c24ca11c1087509620908cd23a9805ab14bdd5e6db8139dcae25cef92df458\": container with ID starting with 43c24ca11c1087509620908cd23a9805ab14bdd5e6db8139dcae25cef92df458 not found: ID does not exist" Oct 30 18:28:36.471895 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:36.471861 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b"] Oct 30 18:28:36.483992 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:36.483963 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-e5d168-566cbc75c6-wv62b"] Oct 30 18:28:36.803754 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:36.803721 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2" path="/var/lib/kubelet/pods/9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2/volumes" Oct 30 18:28:37.829095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.829064 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh"] Oct 30 18:28:37.829756 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.829741 5232 cpu_manager.go:414] "containerMap: removing container" podUID="9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2" containerName="payload-processing" Oct 30 18:28:37.829807 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.829758 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2" containerName="payload-processing" Oct 30 18:28:37.829862 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.829852 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c2d4e2f-17e3-4fa7-a9cc-11c74c99d4d2" containerName="payload-processing" Oct 30 18:28:37.834738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.834717 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh" Oct 30 18:28:37.853525 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.853486 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh"] Oct 30 18:28:37.861417 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.861390 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:28:37.861600 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.861552 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:28:37.882983 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.882954 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95"] Oct 30 18:28:37.888689 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.888662 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95" Oct 30 18:28:37.892234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.892198 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1212add5-16fa-46d3-8f47-ed01ad973d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh\" (UID: \"1212add5-16fa-46d3-8f47-ed01ad973d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh" Oct 30 18:28:37.892412 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.892391 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xks6g\" (UniqueName: \"kubernetes.io/projected/1212add5-16fa-46d3-8f47-ed01ad973d6c-kube-api-access-xks6g\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh\" (UID: \"1212add5-16fa-46d3-8f47-ed01ad973d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh" Oct 30 18:28:37.892487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.892454 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1212add5-16fa-46d3-8f47-ed01ad973d6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh\" (UID: \"1212add5-16fa-46d3-8f47-ed01ad973d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh" Oct 30 18:28:37.908207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.908170 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95"] Oct 30 18:28:37.994131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.994094 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2696076d-3cc9-4859-bf02-25ab6c93ce79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30ae5c7c-hfh95\" (UID: \"2696076d-3cc9-4859-bf02-25ab6c93ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95" Oct 30 18:28:37.994300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.994148 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xks6g\" (UniqueName: \"kubernetes.io/projected/1212add5-16fa-46d3-8f47-ed01ad973d6c-kube-api-access-xks6g\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh\" (UID: \"1212add5-16fa-46d3-8f47-ed01ad973d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh" Oct 30 18:28:37.994300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.994237 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1212add5-16fa-46d3-8f47-ed01ad973d6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh\" (UID: \"1212add5-16fa-46d3-8f47-ed01ad973d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh" Oct 30 18:28:37.994408 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.994387 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2696076d-3cc9-4859-bf02-25ab6c93ce79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30ae5c7c-hfh95\" (UID: \"2696076d-3cc9-4859-bf02-25ab6c93ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95" Oct 30 18:28:37.994458 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.994443 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kv7fz\" (UniqueName: \"kubernetes.io/projected/2696076d-3cc9-4859-bf02-25ab6c93ce79-kube-api-access-kv7fz\") pod \"maas-api-revoke-sub-30ae5c7c-hfh95\" (UID: \"2696076d-3cc9-4859-bf02-25ab6c93ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95" Oct 30 18:28:37.994605 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.994586 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1212add5-16fa-46d3-8f47-ed01ad973d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh\" (UID: \"1212add5-16fa-46d3-8f47-ed01ad973d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh" Oct 30 18:28:37.995180 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.995161 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1212add5-16fa-46d3-8f47-ed01ad973d6c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh\" (UID: \"1212add5-16fa-46d3-8f47-ed01ad973d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh" Oct 30 18:28:37.996742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:37.996725 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1212add5-16fa-46d3-8f47-ed01ad973d6c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh\" (UID: \"1212add5-16fa-46d3-8f47-ed01ad973d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh" Oct 30 18:28:38.016975 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.016952 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xks6g\" (UniqueName: \"kubernetes.io/projected/1212add5-16fa-46d3-8f47-ed01ad973d6c-kube-api-access-xks6g\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh\" (UID: \"1212add5-16fa-46d3-8f47-ed01ad973d6c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh" Oct 30 18:28:38.095680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.095593 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2696076d-3cc9-4859-bf02-25ab6c93ce79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30ae5c7c-hfh95\" (UID: \"2696076d-3cc9-4859-bf02-25ab6c93ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95" Oct 30 18:28:38.095680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.095671 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kv7fz\" (UniqueName: \"kubernetes.io/projected/2696076d-3cc9-4859-bf02-25ab6c93ce79-kube-api-access-kv7fz\") pod \"maas-api-revoke-sub-30ae5c7c-hfh95\" (UID: \"2696076d-3cc9-4859-bf02-25ab6c93ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95" Oct 30 18:28:38.095865 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.095805 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2696076d-3cc9-4859-bf02-25ab6c93ce79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30ae5c7c-hfh95\" (UID: \"2696076d-3cc9-4859-bf02-25ab6c93ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95" Oct 30 18:28:38.096233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.096212 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2696076d-3cc9-4859-bf02-25ab6c93ce79-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-30ae5c7c-hfh95\" (UID: \"2696076d-3cc9-4859-bf02-25ab6c93ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95" Oct 30 18:28:38.098224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.098200 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2696076d-3cc9-4859-bf02-25ab6c93ce79-maas-api-xks-ca\") pod \"maas-api-revoke-sub-30ae5c7c-hfh95\" (UID: \"2696076d-3cc9-4859-bf02-25ab6c93ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95" Oct 30 18:28:38.115979 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.115952 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kv7fz\" (UniqueName: \"kubernetes.io/projected/2696076d-3cc9-4859-bf02-25ab6c93ce79-kube-api-access-kv7fz\") pod \"maas-api-revoke-sub-30ae5c7c-hfh95\" (UID: \"2696076d-3cc9-4859-bf02-25ab6c93ce79\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95" Oct 30 18:28:38.150586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.150352 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh" Oct 30 18:28:38.207717 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.207490 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95" Oct 30 18:28:38.283073 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.282700 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" Oct 30 18:28:38.343762 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.343729 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" Oct 30 18:28:38.405042 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.403668 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/552542b1-fdd2-417f-88a5-614f087e6228-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552542b1-fdd2-417f-88a5-614f087e6228-plugins-config-volume\") pod \"552542b1-fdd2-417f-88a5-614f087e6228\" (UID: \"552542b1-fdd2-417f-88a5-614f087e6228\") " Oct 30 18:28:38.405042 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.403895 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/771db49d-214d-4901-8885-50e40d3d4d88-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/771db49d-214d-4901-8885-50e40d3d4d88-plugins-config-volume\") pod \"771db49d-214d-4901-8885-50e40d3d4d88\" (UID: \"771db49d-214d-4901-8885-50e40d3d4d88\") " Oct 30 18:28:38.405042 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.404071 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/552542b1-fdd2-417f-88a5-614f087e6228-kube-api-access-xktzg\" (UniqueName: \"kubernetes.io/projected/552542b1-fdd2-417f-88a5-614f087e6228-kube-api-access-xktzg\") pod \"552542b1-fdd2-417f-88a5-614f087e6228\" (UID: \"552542b1-fdd2-417f-88a5-614f087e6228\") " Oct 30 18:28:38.405042 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.404123 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/771db49d-214d-4901-8885-50e40d3d4d88-kube-api-access-gwztx\" (UniqueName: \"kubernetes.io/projected/771db49d-214d-4901-8885-50e40d3d4d88-kube-api-access-gwztx\") pod \"771db49d-214d-4901-8885-50e40d3d4d88\" (UID: \"771db49d-214d-4901-8885-50e40d3d4d88\") " Oct 30 18:28:38.408966 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.408815 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/771db49d-214d-4901-8885-50e40d3d4d88-plugins-config-volume" pod "771db49d-214d-4901-8885-50e40d3d4d88" (UID: "771db49d-214d-4901-8885-50e40d3d4d88"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:38.409301 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.409207 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/771db49d-214d-4901-8885-50e40d3d4d88-kube-api-access-gwztx" pod "771db49d-214d-4901-8885-50e40d3d4d88" (UID: "771db49d-214d-4901-8885-50e40d3d4d88"). InnerVolumeSpecName "kube-api-access-gwztx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:38.409439 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.409408 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/552542b1-fdd2-417f-88a5-614f087e6228-plugins-config-volume" pod "552542b1-fdd2-417f-88a5-614f087e6228" (UID: "552542b1-fdd2-417f-88a5-614f087e6228"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:38.413953 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.413924 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/552542b1-fdd2-417f-88a5-614f087e6228-kube-api-access-xktzg" pod "552542b1-fdd2-417f-88a5-614f087e6228" (UID: "552542b1-fdd2-417f-88a5-614f087e6228"). InnerVolumeSpecName "kube-api-access-xktzg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:38.437402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.437369 5232 generic.go:356] "Generic (PLEG): container finished" podID="552542b1-fdd2-417f-88a5-614f087e6228" containerID="bb8129c4fc7a57e50f8b02a91fe453f809e049c8679ed3accda424d6388d2d89" exitCode=0 Oct 30 18:28:38.437623 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.437447 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" Oct 30 18:28:38.437623 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.437488 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" event={"ID":"552542b1-fdd2-417f-88a5-614f087e6228","Type":"ContainerDied","Data":"bb8129c4fc7a57e50f8b02a91fe453f809e049c8679ed3accda424d6388d2d89"} Oct 30 18:28:38.437623 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.437569 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m" event={"ID":"552542b1-fdd2-417f-88a5-614f087e6228","Type":"ContainerDied","Data":"ab46474ac69b0857f6e7a111388ac3f37c5b20aa2c56cc463422efde06039759"} Oct 30 18:28:38.437623 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.437602 5232 scope.go:122] "RemoveContainer" containerID="bb8129c4fc7a57e50f8b02a91fe453f809e049c8679ed3accda424d6388d2d89" Oct 30 18:28:38.439843 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.439815 5232 generic.go:356] "Generic (PLEG): container finished" podID="771db49d-214d-4901-8885-50e40d3d4d88" containerID="65835ba5475943c5565b97503b07cce5f2c3d989d2660d206d14340a1b7acf4a" exitCode=0 Oct 30 18:28:38.439992 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.439973 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" event={"ID":"771db49d-214d-4901-8885-50e40d3d4d88","Type":"ContainerDied","Data":"65835ba5475943c5565b97503b07cce5f2c3d989d2660d206d14340a1b7acf4a"} Oct 30 18:28:38.440051 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.440027 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" event={"ID":"771db49d-214d-4901-8885-50e40d3d4d88","Type":"ContainerDied","Data":"52d9932d4a14eeb2f7fdf1018c180f3191ad25581c674a984a0a484b578f6af6"} Oct 30 18:28:38.440120 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.440105 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z" Oct 30 18:28:38.463146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.463121 5232 scope.go:122] "RemoveContainer" containerID="bb8129c4fc7a57e50f8b02a91fe453f809e049c8679ed3accda424d6388d2d89" Oct 30 18:28:38.463621 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:28:38.463591 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bb8129c4fc7a57e50f8b02a91fe453f809e049c8679ed3accda424d6388d2d89\": container with ID starting with bb8129c4fc7a57e50f8b02a91fe453f809e049c8679ed3accda424d6388d2d89 not found: ID does not exist" containerID="bb8129c4fc7a57e50f8b02a91fe453f809e049c8679ed3accda424d6388d2d89" Oct 30 18:28:38.463803 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.463771 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bb8129c4fc7a57e50f8b02a91fe453f809e049c8679ed3accda424d6388d2d89"} err="failed to get container status \"bb8129c4fc7a57e50f8b02a91fe453f809e049c8679ed3accda424d6388d2d89\": rpc error: code = NotFound desc = could not find container \"bb8129c4fc7a57e50f8b02a91fe453f809e049c8679ed3accda424d6388d2d89\": container with ID starting with bb8129c4fc7a57e50f8b02a91fe453f809e049c8679ed3accda424d6388d2d89 not found: ID does not exist" Oct 30 18:28:38.463923 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.463911 5232 scope.go:122] "RemoveContainer" containerID="65835ba5475943c5565b97503b07cce5f2c3d989d2660d206d14340a1b7acf4a" Oct 30 18:28:38.475835 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.475817 5232 scope.go:122] "RemoveContainer" containerID="65835ba5475943c5565b97503b07cce5f2c3d989d2660d206d14340a1b7acf4a" Oct 30 18:28:38.476126 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:28:38.476108 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65835ba5475943c5565b97503b07cce5f2c3d989d2660d206d14340a1b7acf4a\": container with ID starting with 65835ba5475943c5565b97503b07cce5f2c3d989d2660d206d14340a1b7acf4a not found: ID does not exist" containerID="65835ba5475943c5565b97503b07cce5f2c3d989d2660d206d14340a1b7acf4a" Oct 30 18:28:38.476203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.476133 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65835ba5475943c5565b97503b07cce5f2c3d989d2660d206d14340a1b7acf4a"} err="failed to get container status \"65835ba5475943c5565b97503b07cce5f2c3d989d2660d206d14340a1b7acf4a\": rpc error: code = NotFound desc = could not find container \"65835ba5475943c5565b97503b07cce5f2c3d989d2660d206d14340a1b7acf4a\": container with ID starting with 65835ba5475943c5565b97503b07cce5f2c3d989d2660d206d14340a1b7acf4a not found: ID does not exist" Oct 30 18:28:38.488449 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:28:38.488418 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1212add5_16fa_46d3_8f47_ed01ad973d6c.slice/crio-51e1b4ffd0921e52fa54b8640866562ab7c477d120df540614c0d24bf2eebc6c WatchSource:0}: Error finding container 51e1b4ffd0921e52fa54b8640866562ab7c477d120df540614c0d24bf2eebc6c: Status 404 returned error can't find the container with id 51e1b4ffd0921e52fa54b8640866562ab7c477d120df540614c0d24bf2eebc6c Oct 30 18:28:38.491009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.490982 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh"] Oct 30 18:28:38.506948 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.506910 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xktzg\" (UniqueName: \"kubernetes.io/projected/552542b1-fdd2-417f-88a5-614f087e6228-kube-api-access-xktzg\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:38.507114 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.507099 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwztx\" (UniqueName: \"kubernetes.io/projected/771db49d-214d-4901-8885-50e40d3d4d88-kube-api-access-gwztx\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:38.507233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.507221 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/552542b1-fdd2-417f-88a5-614f087e6228-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:38.507409 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.507331 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/771db49d-214d-4901-8885-50e40d3d4d88-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:38.512407 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.512381 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z"] Oct 30 18:28:38.516722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.516696 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w6-ef13cc-6d64bb7595-lwb9z"] Oct 30 18:28:38.550614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.550588 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m"] Oct 30 18:28:38.560258 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.560232 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95"] Oct 30 18:28:38.561316 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:28:38.561293 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2696076d_3cc9_4859_bf02_25ab6c93ce79.slice/crio-cf56e072c8406d5753d279e91cc3377bb631b411c875aca73f3f3582a3211484 WatchSource:0}: Error finding container cf56e072c8406d5753d279e91cc3377bb631b411c875aca73f3f3582a3211484: Status 404 returned error can't find the container with id cf56e072c8406d5753d279e91cc3377bb631b411c875aca73f3f3582a3211484 Oct 30 18:28:38.563675 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.563652 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-ef13cc-755485dc8d-gx67m"] Oct 30 18:28:38.803170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.803139 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="552542b1-fdd2-417f-88a5-614f087e6228" path="/var/lib/kubelet/pods/552542b1-fdd2-417f-88a5-614f087e6228/volumes" Oct 30 18:28:38.803593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:38.803578 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="771db49d-214d-4901-8885-50e40d3d4d88" path="/var/lib/kubelet/pods/771db49d-214d-4901-8885-50e40d3d4d88/volumes" Oct 30 18:28:39.452223 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:39.452192 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95" event={"ID":"2696076d-3cc9-4859-bf02-25ab6c93ce79","Type":"ContainerStarted","Data":"ae1d8a6fb4f0b3a0d5d1fb709fd76eda8c155b90cd123db9659dd15de8015eb8"} Oct 30 18:28:39.452693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:39.452247 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95" event={"ID":"2696076d-3cc9-4859-bf02-25ab6c93ce79","Type":"ContainerStarted","Data":"cf56e072c8406d5753d279e91cc3377bb631b411c875aca73f3f3582a3211484"} Oct 30 18:28:39.453892 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:39.453869 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh" event={"ID":"1212add5-16fa-46d3-8f47-ed01ad973d6c","Type":"ContainerStarted","Data":"81357893498d7486f1c0e4f5915aea050ee6a85e19dfd5776e156d40cb4e0c31"} Oct 30 18:28:39.453991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:39.453922 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh" event={"ID":"1212add5-16fa-46d3-8f47-ed01ad973d6c","Type":"ContainerStarted","Data":"51e1b4ffd0921e52fa54b8640866562ab7c477d120df540614c0d24bf2eebc6c"} Oct 30 18:28:39.491891 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:39.491843 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95" podStartSLOduration=2.491829659 podStartE2EDuration="2.491829659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:39.490212203 +0000 UTC m=+1618.806632525" watchObservedRunningTime="2026-10-30 18:28:39.491829659 +0000 UTC m=+1618.808250015" Oct 30 18:28:39.532969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:39.532923 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh" podStartSLOduration=2.532909504 podStartE2EDuration="2.532909504s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:39.529853385 +0000 UTC m=+1618.846273709" watchObservedRunningTime="2026-10-30 18:28:39.532909504 +0000 UTC m=+1618.849329825" Oct 30 18:28:40.464827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:40.464793 5232 generic.go:356] "Generic (PLEG): container finished" podID="2696076d-3cc9-4859-bf02-25ab6c93ce79" containerID="ae1d8a6fb4f0b3a0d5d1fb709fd76eda8c155b90cd123db9659dd15de8015eb8" exitCode=0 Oct 30 18:28:40.465272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:40.464873 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95" event={"ID":"2696076d-3cc9-4859-bf02-25ab6c93ce79","Type":"ContainerDied","Data":"ae1d8a6fb4f0b3a0d5d1fb709fd76eda8c155b90cd123db9659dd15de8015eb8"} Oct 30 18:28:40.466756 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:40.466729 5232 generic.go:356] "Generic (PLEG): container finished" podID="1212add5-16fa-46d3-8f47-ed01ad973d6c" containerID="81357893498d7486f1c0e4f5915aea050ee6a85e19dfd5776e156d40cb4e0c31" exitCode=0 Oct 30 18:28:40.466885 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:40.466771 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh" event={"ID":"1212add5-16fa-46d3-8f47-ed01ad973d6c","Type":"ContainerDied","Data":"81357893498d7486f1c0e4f5915aea050ee6a85e19dfd5776e156d40cb4e0c31"} Oct 30 18:28:41.690517 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.690476 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95" Oct 30 18:28:41.694547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.694522 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh" Oct 30 18:28:41.756250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.756220 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1212add5-16fa-46d3-8f47-ed01ad973d6c-kube-api-access-xks6g\" (UniqueName: \"kubernetes.io/projected/1212add5-16fa-46d3-8f47-ed01ad973d6c-kube-api-access-xks6g\") pod \"1212add5-16fa-46d3-8f47-ed01ad973d6c\" (UID: \"1212add5-16fa-46d3-8f47-ed01ad973d6c\") " Oct 30 18:28:41.756460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.756291 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2696076d-3cc9-4859-bf02-25ab6c93ce79-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2696076d-3cc9-4859-bf02-25ab6c93ce79-maas-api-openshift-ca\") pod \"2696076d-3cc9-4859-bf02-25ab6c93ce79\" (UID: \"2696076d-3cc9-4859-bf02-25ab6c93ce79\") " Oct 30 18:28:41.756460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.756385 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1212add5-16fa-46d3-8f47-ed01ad973d6c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1212add5-16fa-46d3-8f47-ed01ad973d6c-maas-api-xks-ca\") pod \"1212add5-16fa-46d3-8f47-ed01ad973d6c\" (UID: \"1212add5-16fa-46d3-8f47-ed01ad973d6c\") " Oct 30 18:28:41.756658 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.756466 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1212add5-16fa-46d3-8f47-ed01ad973d6c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1212add5-16fa-46d3-8f47-ed01ad973d6c-maas-api-openshift-ca\") pod \"1212add5-16fa-46d3-8f47-ed01ad973d6c\" (UID: \"1212add5-16fa-46d3-8f47-ed01ad973d6c\") " Oct 30 18:28:41.756658 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.756597 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2696076d-3cc9-4859-bf02-25ab6c93ce79-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2696076d-3cc9-4859-bf02-25ab6c93ce79-maas-api-xks-ca\") pod \"2696076d-3cc9-4859-bf02-25ab6c93ce79\" (UID: \"2696076d-3cc9-4859-bf02-25ab6c93ce79\") " Oct 30 18:28:41.756780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.756728 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2696076d-3cc9-4859-bf02-25ab6c93ce79-kube-api-access-kv7fz\" (UniqueName: \"kubernetes.io/projected/2696076d-3cc9-4859-bf02-25ab6c93ce79-kube-api-access-kv7fz\") pod \"2696076d-3cc9-4859-bf02-25ab6c93ce79\" (UID: \"2696076d-3cc9-4859-bf02-25ab6c93ce79\") " Oct 30 18:28:41.756846 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.756777 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2696076d-3cc9-4859-bf02-25ab6c93ce79-maas-api-openshift-ca" pod "2696076d-3cc9-4859-bf02-25ab6c93ce79" (UID: "2696076d-3cc9-4859-bf02-25ab6c93ce79"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:41.756913 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.756851 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1212add5-16fa-46d3-8f47-ed01ad973d6c-maas-api-openshift-ca" pod "1212add5-16fa-46d3-8f47-ed01ad973d6c" (UID: "1212add5-16fa-46d3-8f47-ed01ad973d6c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:41.757417 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.757388 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1212add5-16fa-46d3-8f47-ed01ad973d6c-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:41.757542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.757421 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2696076d-3cc9-4859-bf02-25ab6c93ce79-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:41.759561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.759430 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1212add5-16fa-46d3-8f47-ed01ad973d6c-kube-api-access-xks6g" pod "1212add5-16fa-46d3-8f47-ed01ad973d6c" (UID: "1212add5-16fa-46d3-8f47-ed01ad973d6c"). InnerVolumeSpecName "kube-api-access-xks6g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:41.759561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.759486 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2696076d-3cc9-4859-bf02-25ab6c93ce79-maas-api-xks-ca" pod "2696076d-3cc9-4859-bf02-25ab6c93ce79" (UID: "2696076d-3cc9-4859-bf02-25ab6c93ce79"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:41.759696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.759572 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1212add5-16fa-46d3-8f47-ed01ad973d6c-maas-api-xks-ca" pod "1212add5-16fa-46d3-8f47-ed01ad973d6c" (UID: "1212add5-16fa-46d3-8f47-ed01ad973d6c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:41.759818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.759798 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2696076d-3cc9-4859-bf02-25ab6c93ce79-kube-api-access-kv7fz" pod "2696076d-3cc9-4859-bf02-25ab6c93ce79" (UID: "2696076d-3cc9-4859-bf02-25ab6c93ce79"). InnerVolumeSpecName "kube-api-access-kv7fz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:41.859391 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.859359 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1212add5-16fa-46d3-8f47-ed01ad973d6c-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:41.859391 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.859388 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2696076d-3cc9-4859-bf02-25ab6c93ce79-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:41.859622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.859400 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kv7fz\" (UniqueName: \"kubernetes.io/projected/2696076d-3cc9-4859-bf02-25ab6c93ce79-kube-api-access-kv7fz\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:41.859622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:41.859413 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xks6g\" (UniqueName: \"kubernetes.io/projected/1212add5-16fa-46d3-8f47-ed01ad973d6c-kube-api-access-xks6g\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:42.373380 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:42.373347 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9"] Oct 30 18:28:42.373879 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:42.373732 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" podUID="9c3eac53-5e79-43d3-8965-d45420dd403b" containerName="istio-proxy" containerID="cri-o://dad2532a7de8909d7b098a148fde8fa42f2b98acc783b10bbee6992981d6a09c" gracePeriod=30 Oct 30 18:28:42.383671 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:42.383614 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6"] Oct 30 18:28:42.383857 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:42.383832 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" podUID="cee56829-eea8-4aa6-b006-1bb43b731693" containerName="istio-proxy" containerID="cri-o://61b7dcaa069062171dc1b3a45fd7d0ecc9b6d0e9f0a6d1c59300f3271f14dd1c" gracePeriod=30 Oct 30 18:28:42.409110 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:42.409085 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8"] Oct 30 18:28:42.409290 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:42.409257 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" containerID="cri-o://2fc65f582b1aa534c940310ef99259fcbdf80cbf2e01a21b208668f05f3c5855" gracePeriod=30 Oct 30 18:28:42.578194 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:42.578152 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95" event={"ID":"2696076d-3cc9-4859-bf02-25ab6c93ce79","Type":"ContainerDied","Data":"cf56e072c8406d5753d279e91cc3377bb631b411c875aca73f3f3582a3211484"} Oct 30 18:28:42.578406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:42.578222 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cf56e072c8406d5753d279e91cc3377bb631b411c875aca73f3f3582a3211484" Oct 30 18:28:42.578406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:42.578310 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95" Oct 30 18:28:42.597346 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:42.597000 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh" event={"ID":"1212add5-16fa-46d3-8f47-ed01ad973d6c","Type":"ContainerDied","Data":"51e1b4ffd0921e52fa54b8640866562ab7c477d120df540614c0d24bf2eebc6c"} Oct 30 18:28:42.597346 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:42.597066 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="51e1b4ffd0921e52fa54b8640866562ab7c477d120df540614c0d24bf2eebc6c" Oct 30 18:28:42.597346 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:42.597139 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh" Oct 30 18:28:43.197022 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.196982 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs"] Oct 30 18:28:43.197654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.197639 5232 cpu_manager.go:414] "containerMap: removing container" podUID="552542b1-fdd2-417f-88a5-614f087e6228" containerName="payload-processing" Oct 30 18:28:43.197723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.197656 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="552542b1-fdd2-417f-88a5-614f087e6228" containerName="payload-processing" Oct 30 18:28:43.197723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.197700 5232 cpu_manager.go:414] "containerMap: removing container" podUID="2696076d-3cc9-4859-bf02-25ab6c93ce79" containerName="revoke-keys" Oct 30 18:28:43.197723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.197705 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2696076d-3cc9-4859-bf02-25ab6c93ce79" containerName="revoke-keys" Oct 30 18:28:43.197723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.197718 5232 cpu_manager.go:414] "containerMap: removing container" podUID="771db49d-214d-4901-8885-50e40d3d4d88" containerName="payload-pre-processing" Oct 30 18:28:43.197723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.197723 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="771db49d-214d-4901-8885-50e40d3d4d88" containerName="payload-pre-processing" Oct 30 18:28:43.197907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.197734 5232 cpu_manager.go:414] "containerMap: removing container" podUID="1212add5-16fa-46d3-8f47-ed01ad973d6c" containerName="revoke-keys" Oct 30 18:28:43.197907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.197739 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1212add5-16fa-46d3-8f47-ed01ad973d6c" containerName="revoke-keys" Oct 30 18:28:43.197907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.197845 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="771db49d-214d-4901-8885-50e40d3d4d88" containerName="payload-pre-processing" Oct 30 18:28:43.197907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.197857 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="1212add5-16fa-46d3-8f47-ed01ad973d6c" containerName="revoke-keys" Oct 30 18:28:43.197907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.197866 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="552542b1-fdd2-417f-88a5-614f087e6228" containerName="payload-processing" Oct 30 18:28:43.197907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.197877 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="2696076d-3cc9-4859-bf02-25ab6c93ce79" containerName="revoke-keys" Oct 30 18:28:43.201941 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.201924 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs" Oct 30 18:28:43.224668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.223972 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:28:43.224905 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.224887 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:28:43.225836 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.225807 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs"] Oct 30 18:28:43.298154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.298123 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f1cecd6-c961-445c-955e-5a8f8b4f3168-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs\" (UID: \"4f1cecd6-c961-445c-955e-5a8f8b4f3168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs" Oct 30 18:28:43.298322 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.298178 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bvps\" (UniqueName: \"kubernetes.io/projected/4f1cecd6-c961-445c-955e-5a8f8b4f3168-kube-api-access-9bvps\") pod \"maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs\" (UID: \"4f1cecd6-c961-445c-955e-5a8f8b4f3168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs" Oct 30 18:28:43.298381 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.298323 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f1cecd6-c961-445c-955e-5a8f8b4f3168-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs\" (UID: \"4f1cecd6-c961-445c-955e-5a8f8b4f3168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs" Oct 30 18:28:43.399765 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.399728 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f1cecd6-c961-445c-955e-5a8f8b4f3168-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs\" (UID: \"4f1cecd6-c961-445c-955e-5a8f8b4f3168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs" Oct 30 18:28:43.399962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.399811 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bvps\" (UniqueName: \"kubernetes.io/projected/4f1cecd6-c961-445c-955e-5a8f8b4f3168-kube-api-access-9bvps\") pod \"maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs\" (UID: \"4f1cecd6-c961-445c-955e-5a8f8b4f3168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs" Oct 30 18:28:43.399962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.399930 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f1cecd6-c961-445c-955e-5a8f8b4f3168-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs\" (UID: \"4f1cecd6-c961-445c-955e-5a8f8b4f3168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs" Oct 30 18:28:43.400700 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.400660 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f1cecd6-c961-445c-955e-5a8f8b4f3168-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs\" (UID: \"4f1cecd6-c961-445c-955e-5a8f8b4f3168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs" Oct 30 18:28:43.402367 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.402337 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f1cecd6-c961-445c-955e-5a8f8b4f3168-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs\" (UID: \"4f1cecd6-c961-445c-955e-5a8f8b4f3168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs" Oct 30 18:28:43.424056 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.424020 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bvps\" (UniqueName: \"kubernetes.io/projected/4f1cecd6-c961-445c-955e-5a8f8b4f3168-kube-api-access-9bvps\") pod \"maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs\" (UID: \"4f1cecd6-c961-445c-955e-5a8f8b4f3168\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs" Oct 30 18:28:43.516542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.516489 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs" Oct 30 18:28:43.842519 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:43.842477 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs"] Oct 30 18:28:43.843137 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:28:43.842804 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4f1cecd6_c961_445c_955e_5a8f8b4f3168.slice/crio-e2be911483254dfe110eaa5020b034add3b1fa1a9c8a8ebadc566dfeaa33faa2 WatchSource:0}: Error finding container e2be911483254dfe110eaa5020b034add3b1fa1a9c8a8ebadc566dfeaa33faa2: Status 404 returned error can't find the container with id e2be911483254dfe110eaa5020b034add3b1fa1a9c8a8ebadc566dfeaa33faa2 Oct 30 18:28:44.425564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:44.423788 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k" Oct 30 18:28:44.652371 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:44.652319 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs" event={"ID":"4f1cecd6-c961-445c-955e-5a8f8b4f3168","Type":"ContainerStarted","Data":"22dd2addab4a7884dfb549cc122bba839e7ff56ca9b588b4196134ec562fa8d2"} Oct 30 18:28:44.652832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:44.652805 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs" event={"ID":"4f1cecd6-c961-445c-955e-5a8f8b4f3168","Type":"ContainerStarted","Data":"e2be911483254dfe110eaa5020b034add3b1fa1a9c8a8ebadc566dfeaa33faa2"} Oct 30 18:28:44.703872 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:44.703798 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs" podStartSLOduration=1.703772652 podStartE2EDuration="1.703772652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:28:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:28:44.694887545 +0000 UTC m=+1624.011307867" watchObservedRunningTime="2026-10-30 18:28:44.703772652 +0000 UTC m=+1624.020192974" Oct 30 18:28:45.661604 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:45.661565 5232 generic.go:356] "Generic (PLEG): container finished" podID="4f1cecd6-c961-445c-955e-5a8f8b4f3168" containerID="22dd2addab4a7884dfb549cc122bba839e7ff56ca9b588b4196134ec562fa8d2" exitCode=0 Oct 30 18:28:45.662009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:45.661613 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs" event={"ID":"4f1cecd6-c961-445c-955e-5a8f8b4f3168","Type":"ContainerDied","Data":"22dd2addab4a7884dfb549cc122bba839e7ff56ca9b588b4196134ec562fa8d2"} Oct 30 18:28:46.831639 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:46.831611 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs" Oct 30 18:28:46.987187 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:46.987157 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4f1cecd6-c961-445c-955e-5a8f8b4f3168-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f1cecd6-c961-445c-955e-5a8f8b4f3168-maas-api-xks-ca\") pod \"4f1cecd6-c961-445c-955e-5a8f8b4f3168\" (UID: \"4f1cecd6-c961-445c-955e-5a8f8b4f3168\") " Oct 30 18:28:46.987344 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:46.987269 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4f1cecd6-c961-445c-955e-5a8f8b4f3168-kube-api-access-9bvps\" (UniqueName: \"kubernetes.io/projected/4f1cecd6-c961-445c-955e-5a8f8b4f3168-kube-api-access-9bvps\") pod \"4f1cecd6-c961-445c-955e-5a8f8b4f3168\" (UID: \"4f1cecd6-c961-445c-955e-5a8f8b4f3168\") " Oct 30 18:28:46.987434 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:46.987346 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4f1cecd6-c961-445c-955e-5a8f8b4f3168-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f1cecd6-c961-445c-955e-5a8f8b4f3168-maas-api-openshift-ca\") pod \"4f1cecd6-c961-445c-955e-5a8f8b4f3168\" (UID: \"4f1cecd6-c961-445c-955e-5a8f8b4f3168\") " Oct 30 18:28:46.987767 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:46.987739 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4f1cecd6-c961-445c-955e-5a8f8b4f3168-maas-api-openshift-ca" pod "4f1cecd6-c961-445c-955e-5a8f8b4f3168" (UID: "4f1cecd6-c961-445c-955e-5a8f8b4f3168"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:46.988112 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:46.988095 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4f1cecd6-c961-445c-955e-5a8f8b4f3168-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:46.989452 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:46.989428 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4f1cecd6-c961-445c-955e-5a8f8b4f3168-maas-api-xks-ca" pod "4f1cecd6-c961-445c-955e-5a8f8b4f3168" (UID: "4f1cecd6-c961-445c-955e-5a8f8b4f3168"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:46.989572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:46.989556 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4f1cecd6-c961-445c-955e-5a8f8b4f3168-kube-api-access-9bvps" pod "4f1cecd6-c961-445c-955e-5a8f8b4f3168" (UID: "4f1cecd6-c961-445c-955e-5a8f8b4f3168"). InnerVolumeSpecName "kube-api-access-9bvps". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:47.089375 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.089341 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bvps\" (UniqueName: \"kubernetes.io/projected/4f1cecd6-c961-445c-955e-5a8f8b4f3168-kube-api-access-9bvps\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.089375 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.089373 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4f1cecd6-c961-445c-955e-5a8f8b4f3168-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.680691 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.680664 5232 generic.go:356] "Generic (PLEG): container finished" podID="9c3eac53-5e79-43d3-8965-d45420dd403b" containerID="dad2532a7de8909d7b098a148fde8fa42f2b98acc783b10bbee6992981d6a09c" exitCode=0 Oct 30 18:28:47.680813 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.680798 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" event={"ID":"9c3eac53-5e79-43d3-8965-d45420dd403b","Type":"ContainerDied","Data":"dad2532a7de8909d7b098a148fde8fa42f2b98acc783b10bbee6992981d6a09c"} Oct 30 18:28:47.682435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.682410 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs" Oct 30 18:28:47.682684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.682419 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs" event={"ID":"4f1cecd6-c961-445c-955e-5a8f8b4f3168","Type":"ContainerDied","Data":"e2be911483254dfe110eaa5020b034add3b1fa1a9c8a8ebadc566dfeaa33faa2"} Oct 30 18:28:47.682684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.682535 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e2be911483254dfe110eaa5020b034add3b1fa1a9c8a8ebadc566dfeaa33faa2" Oct 30 18:28:47.685341 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.685130 5232 generic.go:356] "Generic (PLEG): container finished" podID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerID="2fc65f582b1aa534c940310ef99259fcbdf80cbf2e01a21b208668f05f3c5855" exitCode=0 Oct 30 18:28:47.685341 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.685285 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" event={"ID":"6727049d-37d3-47f0-bf07-d63b6bf9aab4","Type":"ContainerDied","Data":"2fc65f582b1aa534c940310ef99259fcbdf80cbf2e01a21b208668f05f3c5855"} Oct 30 18:28:47.687688 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.687553 5232 generic.go:356] "Generic (PLEG): container finished" podID="cee56829-eea8-4aa6-b006-1bb43b731693" containerID="61b7dcaa069062171dc1b3a45fd7d0ecc9b6d0e9f0a6d1c59300f3271f14dd1c" exitCode=0 Oct 30 18:28:47.687688 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.687635 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" event={"ID":"cee56829-eea8-4aa6-b006-1bb43b731693","Type":"ContainerDied","Data":"61b7dcaa069062171dc1b3a45fd7d0ecc9b6d0e9f0a6d1c59300f3271f14dd1c"} Oct 30 18:28:47.735545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.731665 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:28:47.798369 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.798335 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:28:47.802608 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.802584 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee56829-eea8-4aa6-b006-1bb43b731693-istio-token\" (UniqueName: \"kubernetes.io/projected/cee56829-eea8-4aa6-b006-1bb43b731693-istio-token\") pod \"cee56829-eea8-4aa6-b006-1bb43b731693\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " Oct 30 18:28:47.802769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.802614 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cee56829-eea8-4aa6-b006-1bb43b731693-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee56829-eea8-4aa6-b006-1bb43b731693-istiod-ca-cert\") pod \"cee56829-eea8-4aa6-b006-1bb43b731693\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " Oct 30 18:28:47.802769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.802681 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-workload-certs\") pod \"9c3eac53-5e79-43d3-8965-d45420dd403b\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " Oct 30 18:28:47.802769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.802713 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-credential-socket\") pod \"cee56829-eea8-4aa6-b006-1bb43b731693\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " Oct 30 18:28:47.802769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.802744 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-envoy\") pod \"9c3eac53-5e79-43d3-8965-d45420dd403b\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " Oct 30 18:28:47.802931 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.802791 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-istio-envoy\") pod \"cee56829-eea8-4aa6-b006-1bb43b731693\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " Oct 30 18:28:47.802931 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.802826 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-workload-socket\") pod \"9c3eac53-5e79-43d3-8965-d45420dd403b\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " Oct 30 18:28:47.802931 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.802863 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-workload-certs\") pod \"cee56829-eea8-4aa6-b006-1bb43b731693\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " Oct 30 18:28:47.802931 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.802896 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-workload-socket\") pod \"cee56829-eea8-4aa6-b006-1bb43b731693\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " Oct 30 18:28:47.802931 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.802921 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-podinfo\") pod \"9c3eac53-5e79-43d3-8965-d45420dd403b\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " Oct 30 18:28:47.803160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.802954 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-istio-data\") pod \"cee56829-eea8-4aa6-b006-1bb43b731693\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " Oct 30 18:28:47.803160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.802976 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-credential-socket\") pod \"9c3eac53-5e79-43d3-8965-d45420dd403b\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " Oct 30 18:28:47.803160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.803023 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cee56829-eea8-4aa6-b006-1bb43b731693-kube-api-access-cgddb\" (UniqueName: \"kubernetes.io/projected/cee56829-eea8-4aa6-b006-1bb43b731693-kube-api-access-cgddb\") pod \"cee56829-eea8-4aa6-b006-1bb43b731693\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " Oct 30 18:28:47.803160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.803047 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c3eac53-5e79-43d3-8965-d45420dd403b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3eac53-5e79-43d3-8965-d45420dd403b-istiod-ca-cert\") pod \"9c3eac53-5e79-43d3-8965-d45420dd403b\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " Oct 30 18:28:47.803160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.803100 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cee56829-eea8-4aa6-b006-1bb43b731693-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee56829-eea8-4aa6-b006-1bb43b731693-istio-podinfo\") pod \"cee56829-eea8-4aa6-b006-1bb43b731693\" (UID: \"cee56829-eea8-4aa6-b006-1bb43b731693\") " Oct 30 18:28:47.803316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.803171 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3eac53-5e79-43d3-8965-d45420dd403b-kube-api-access-7xq4n\" (UniqueName: \"kubernetes.io/projected/9c3eac53-5e79-43d3-8965-d45420dd403b-kube-api-access-7xq4n\") pod \"9c3eac53-5e79-43d3-8965-d45420dd403b\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " Oct 30 18:28:47.803316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.803213 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-data\") pod \"9c3eac53-5e79-43d3-8965-d45420dd403b\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " Oct 30 18:28:47.803316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.803247 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-token\" (UniqueName: \"kubernetes.io/projected/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-token\") pod \"9c3eac53-5e79-43d3-8965-d45420dd403b\" (UID: \"9c3eac53-5e79-43d3-8965-d45420dd403b\") " Oct 30 18:28:47.804021 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.803858 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-workload-certs" pod "9c3eac53-5e79-43d3-8965-d45420dd403b" (UID: "9c3eac53-5e79-43d3-8965-d45420dd403b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.804021 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.803904 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cee56829-eea8-4aa6-b006-1bb43b731693-istiod-ca-cert" pod "cee56829-eea8-4aa6-b006-1bb43b731693" (UID: "cee56829-eea8-4aa6-b006-1bb43b731693"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:47.804021 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.803958 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-credential-socket" pod "cee56829-eea8-4aa6-b006-1bb43b731693" (UID: "cee56829-eea8-4aa6-b006-1bb43b731693"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.804253 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.804232 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-istio-data" pod "cee56829-eea8-4aa6-b006-1bb43b731693" (UID: "cee56829-eea8-4aa6-b006-1bb43b731693"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.805059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.804934 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-credential-socket" pod "9c3eac53-5e79-43d3-8965-d45420dd403b" (UID: "9c3eac53-5e79-43d3-8965-d45420dd403b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.806223 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.806198 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-workload-socket" pod "cee56829-eea8-4aa6-b006-1bb43b731693" (UID: "cee56829-eea8-4aa6-b006-1bb43b731693"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.806486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.806460 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-workload-socket" pod "9c3eac53-5e79-43d3-8965-d45420dd403b" (UID: "9c3eac53-5e79-43d3-8965-d45420dd403b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.806745 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.806721 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-workload-certs" pod "cee56829-eea8-4aa6-b006-1bb43b731693" (UID: "cee56829-eea8-4aa6-b006-1bb43b731693"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.807029 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.807005 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c3eac53-5e79-43d3-8965-d45420dd403b-istiod-ca-cert" pod "9c3eac53-5e79-43d3-8965-d45420dd403b" (UID: "9c3eac53-5e79-43d3-8965-d45420dd403b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:47.807439 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.807400 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-data" pod "9c3eac53-5e79-43d3-8965-d45420dd403b" (UID: "9c3eac53-5e79-43d3-8965-d45420dd403b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.808916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.808885 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee56829-eea8-4aa6-b006-1bb43b731693-istio-token" pod "cee56829-eea8-4aa6-b006-1bb43b731693" (UID: "cee56829-eea8-4aa6-b006-1bb43b731693"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:47.821988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.821921 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cee56829-eea8-4aa6-b006-1bb43b731693-kube-api-access-cgddb" pod "cee56829-eea8-4aa6-b006-1bb43b731693" (UID: "cee56829-eea8-4aa6-b006-1bb43b731693"). InnerVolumeSpecName "kube-api-access-cgddb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:47.821988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.821970 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-podinfo" pod "9c3eac53-5e79-43d3-8965-d45420dd403b" (UID: "9c3eac53-5e79-43d3-8965-d45420dd403b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:47.822136 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.822058 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cee56829-eea8-4aa6-b006-1bb43b731693-istio-podinfo" pod "cee56829-eea8-4aa6-b006-1bb43b731693" (UID: "cee56829-eea8-4aa6-b006-1bb43b731693"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:47.822177 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.822159 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-envoy" pod "9c3eac53-5e79-43d3-8965-d45420dd403b" (UID: "9c3eac53-5e79-43d3-8965-d45420dd403b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.822279 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.822262 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-token" pod "9c3eac53-5e79-43d3-8965-d45420dd403b" (UID: "9c3eac53-5e79-43d3-8965-d45420dd403b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:47.822386 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.822368 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-istio-envoy" pod "cee56829-eea8-4aa6-b006-1bb43b731693" (UID: "cee56829-eea8-4aa6-b006-1bb43b731693"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.824118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.824092 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c3eac53-5e79-43d3-8965-d45420dd403b-kube-api-access-7xq4n" pod "9c3eac53-5e79-43d3-8965-d45420dd403b" (UID: "9c3eac53-5e79-43d3-8965-d45420dd403b"). InnerVolumeSpecName "kube-api-access-7xq4n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:47.869998 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.869970 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:28:47.904888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.904816 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-envoy\") pod \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " Oct 30 18:28:47.905055 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.904890 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-credential-socket\") pod \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " Oct 30 18:28:47.905055 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.904982 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6727049d-37d3-47f0-bf07-d63b6bf9aab4-kube-api-access-grt9q\" (UniqueName: \"kubernetes.io/projected/6727049d-37d3-47f0-bf07-d63b6bf9aab4-kube-api-access-grt9q\") pod \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " Oct 30 18:28:47.905055 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.905032 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istiod-ca-cert\") pod \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " Oct 30 18:28:47.905282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.905066 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-data\") pod \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " Oct 30 18:28:47.905282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.905159 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-workload-socket\") pod \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " Oct 30 18:28:47.905282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.905178 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-credential-socket" pod "6727049d-37d3-47f0-bf07-d63b6bf9aab4" (UID: "6727049d-37d3-47f0-bf07-d63b6bf9aab4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.905282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.905264 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-token\" (UniqueName: \"kubernetes.io/projected/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-token\") pod \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " Oct 30 18:28:47.905590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.905315 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-podinfo\") pod \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " Oct 30 18:28:47.905590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.905367 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-workload-certs\") pod \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\" (UID: \"6727049d-37d3-47f0-bf07-d63b6bf9aab4\") " Oct 30 18:28:47.905590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.905394 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-data" pod "6727049d-37d3-47f0-bf07-d63b6bf9aab4" (UID: "6727049d-37d3-47f0-bf07-d63b6bf9aab4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.905590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.905414 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istiod-ca-cert" pod "6727049d-37d3-47f0-bf07-d63b6bf9aab4" (UID: "6727049d-37d3-47f0-bf07-d63b6bf9aab4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:47.905590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.905500 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-workload-socket" pod "6727049d-37d3-47f0-bf07-d63b6bf9aab4" (UID: "6727049d-37d3-47f0-bf07-d63b6bf9aab4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.905934 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.905698 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-workload-certs" pod "6727049d-37d3-47f0-bf07-d63b6bf9aab4" (UID: "6727049d-37d3-47f0-bf07-d63b6bf9aab4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.907119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907094 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cee56829-eea8-4aa6-b006-1bb43b731693-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907123 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7xq4n\" (UniqueName: \"kubernetes.io/projected/9c3eac53-5e79-43d3-8965-d45420dd403b-kube-api-access-7xq4n\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907139 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907153 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907167 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907179 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907191 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907204 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cee56829-eea8-4aa6-b006-1bb43b731693-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907216 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cee56829-eea8-4aa6-b006-1bb43b731693-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907228 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907241 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907254 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907268 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907281 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907294 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907308 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907321 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907334 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9c3eac53-5e79-43d3-8965-d45420dd403b-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907347 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cee56829-eea8-4aa6-b006-1bb43b731693-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907360 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9c3eac53-5e79-43d3-8965-d45420dd403b-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907375 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgddb\" (UniqueName: \"kubernetes.io/projected/cee56829-eea8-4aa6-b006-1bb43b731693-kube-api-access-cgddb\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907392 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.907827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.907406 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9c3eac53-5e79-43d3-8965-d45420dd403b-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:47.908845 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.908821 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6727049d-37d3-47f0-bf07-d63b6bf9aab4-kube-api-access-grt9q" pod "6727049d-37d3-47f0-bf07-d63b6bf9aab4" (UID: "6727049d-37d3-47f0-bf07-d63b6bf9aab4"). InnerVolumeSpecName "kube-api-access-grt9q". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:47.908950 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.908878 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-envoy" pod "6727049d-37d3-47f0-bf07-d63b6bf9aab4" (UID: "6727049d-37d3-47f0-bf07-d63b6bf9aab4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:28:47.908950 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.908891 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-podinfo" pod "6727049d-37d3-47f0-bf07-d63b6bf9aab4" (UID: "6727049d-37d3-47f0-bf07-d63b6bf9aab4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:28:47.909753 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.909703 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-token" pod "6727049d-37d3-47f0-bf07-d63b6bf9aab4" (UID: "6727049d-37d3-47f0-bf07-d63b6bf9aab4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:47.930820 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:47.929419 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t"] Oct 30 18:28:48.008271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.008243 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grt9q\" (UniqueName: \"kubernetes.io/projected/6727049d-37d3-47f0-bf07-d63b6bf9aab4-kube-api-access-grt9q\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.008271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.008271 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.008271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.008280 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.008494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.008294 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6727049d-37d3-47f0-bf07-d63b6bf9aab4-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:48.697564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.697503 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" Oct 30 18:28:48.697564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.697539 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8" event={"ID":"6727049d-37d3-47f0-bf07-d63b6bf9aab4","Type":"ContainerDied","Data":"3b6c42222f8b8cfa5e930280cf8eed82dee1441ebb83dcf7be76eaafbf157d24"} Oct 30 18:28:48.697831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.697618 5232 scope.go:122] "RemoveContainer" containerID="2fc65f582b1aa534c940310ef99259fcbdf80cbf2e01a21b208668f05f3c5855" Oct 30 18:28:48.699260 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.699234 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" Oct 30 18:28:48.699385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.699257 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6" event={"ID":"cee56829-eea8-4aa6-b006-1bb43b731693","Type":"ContainerDied","Data":"170d08981f8d4e161d1e0f61ca85dba5346fe35dedb5bf6bb3faffb89b3230c0"} Oct 30 18:28:48.701051 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.701030 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" Oct 30 18:28:48.701136 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.701063 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9" event={"ID":"9c3eac53-5e79-43d3-8965-d45420dd403b","Type":"ContainerDied","Data":"18946b6b3de242245d86f2aebd436bdec2d1ad113836242ccc70d4eeb5ab36b9"} Oct 30 18:28:48.710034 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.710015 5232 scope.go:122] "RemoveContainer" containerID="61b7dcaa069062171dc1b3a45fd7d0ecc9b6d0e9f0a6d1c59300f3271f14dd1c" Oct 30 18:28:48.722233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.722079 5232 scope.go:122] "RemoveContainer" containerID="dad2532a7de8909d7b098a148fde8fa42f2b98acc783b10bbee6992981d6a09c" Oct 30 18:28:48.761919 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.761892 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6"] Oct 30 18:28:48.774176 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.774151 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-wtpp6"] Oct 30 18:28:48.803235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.803066 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cee56829-eea8-4aa6-b006-1bb43b731693" path="/var/lib/kubelet/pods/cee56829-eea8-4aa6-b006-1bb43b731693/volumes" Oct 30 18:28:48.807138 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.807118 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9"] Oct 30 18:28:48.812297 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.812277 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-9nnn9"] Oct 30 18:28:48.846417 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.846365 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8"] Oct 30 18:28:48.849223 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:48.849180 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w0-e5d168-openshift-default-7f9f9479ff-rl8m8"] Oct 30 18:28:50.844589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:50.843792 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" path="/var/lib/kubelet/pods/6727049d-37d3-47f0-bf07-d63b6bf9aab4/volumes" Oct 30 18:28:50.844589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:50.844547 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c3eac53-5e79-43d3-8965-d45420dd403b" path="/var/lib/kubelet/pods/9c3eac53-5e79-43d3-8965-d45420dd403b/volumes" Oct 30 18:28:54.335455 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:54.335420 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh"] Oct 30 18:28:54.345466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:54.345438 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b2b596-tdqwh"] Oct 30 18:28:54.527340 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:54.527308 5232 scope.go:122] "RemoveContainer" containerID="c91ddb1d1cc92bcc4a5950e1faffadaa0732735c9d435738715aaef6c4f229ec" Oct 30 18:28:54.807770 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:54.807738 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1212add5-16fa-46d3-8f47-ed01ad973d6c" path="/var/lib/kubelet/pods/1212add5-16fa-46d3-8f47-ed01ad973d6c/volumes" Oct 30 18:28:55.974593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:55.971808 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc"] Oct 30 18:28:55.974593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:55.972773 5232 cpu_manager.go:414] "containerMap: removing container" podUID="cee56829-eea8-4aa6-b006-1bb43b731693" containerName="istio-proxy" Oct 30 18:28:55.974593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:55.972795 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cee56829-eea8-4aa6-b006-1bb43b731693" containerName="istio-proxy" Oct 30 18:28:55.974593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:55.972822 5232 cpu_manager.go:414] "containerMap: removing container" podUID="4f1cecd6-c961-445c-955e-5a8f8b4f3168" containerName="revoke-keys" Oct 30 18:28:55.974593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:55.972829 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4f1cecd6-c961-445c-955e-5a8f8b4f3168" containerName="revoke-keys" Oct 30 18:28:55.974593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:55.972855 5232 cpu_manager.go:414] "containerMap: removing container" podUID="9c3eac53-5e79-43d3-8965-d45420dd403b" containerName="istio-proxy" Oct 30 18:28:55.974593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:55.972863 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c3eac53-5e79-43d3-8965-d45420dd403b" containerName="istio-proxy" Oct 30 18:28:55.974593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:55.972898 5232 cpu_manager.go:414] "containerMap: removing container" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" Oct 30 18:28:55.974593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:55.972905 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" Oct 30 18:28:55.974593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:55.973037 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="cee56829-eea8-4aa6-b006-1bb43b731693" containerName="istio-proxy" Oct 30 18:28:55.974593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:55.973052 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="4f1cecd6-c961-445c-955e-5a8f8b4f3168" containerName="revoke-keys" Oct 30 18:28:55.974593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:55.973064 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c3eac53-5e79-43d3-8965-d45420dd403b" containerName="istio-proxy" Oct 30 18:28:55.974593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:55.973080 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="6727049d-37d3-47f0-bf07-d63b6bf9aab4" containerName="istio-proxy" Oct 30 18:28:56.001343 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.001308 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc" Oct 30 18:28:56.001666 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.001633 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc"] Oct 30 18:28:56.006219 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.006190 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d"] Oct 30 18:28:56.016427 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.014262 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d" Oct 30 18:28:56.045732 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.045362 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d"] Oct 30 18:28:56.046477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.046456 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:28:56.046806 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.046790 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:28:56.135873 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.135829 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47820c98-v8h9d\" (UID: \"e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d" Oct 30 18:28:56.136058 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.135957 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75673761-hlxbc\" (UID: \"a9b2c9fa-68ce-4152-ba7a-7440e1274eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc" Oct 30 18:28:56.136147 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.136064 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47820c98-v8h9d\" (UID: \"e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d" Oct 30 18:28:56.136147 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.136108 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qx2b2\" (UniqueName: \"kubernetes.io/projected/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-kube-api-access-qx2b2\") pod \"maas-api-revoke-sub-47820c98-v8h9d\" (UID: \"e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d" Oct 30 18:28:56.136233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.136183 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75673761-hlxbc\" (UID: \"a9b2c9fa-68ce-4152-ba7a-7440e1274eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc" Oct 30 18:28:56.136277 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.136241 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfd46\" (UniqueName: \"kubernetes.io/projected/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-kube-api-access-mfd46\") pod \"maas-api-revoke-sub-75673761-hlxbc\" (UID: \"a9b2c9fa-68ce-4152-ba7a-7440e1274eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc" Oct 30 18:28:56.237031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.237001 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47820c98-v8h9d\" (UID: \"e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d" Oct 30 18:28:56.237206 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.237034 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qx2b2\" (UniqueName: \"kubernetes.io/projected/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-kube-api-access-qx2b2\") pod \"maas-api-revoke-sub-47820c98-v8h9d\" (UID: \"e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d" Oct 30 18:28:56.237251 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.237232 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75673761-hlxbc\" (UID: \"a9b2c9fa-68ce-4152-ba7a-7440e1274eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc" Oct 30 18:28:56.237375 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.237344 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfd46\" (UniqueName: \"kubernetes.io/projected/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-kube-api-access-mfd46\") pod \"maas-api-revoke-sub-75673761-hlxbc\" (UID: \"a9b2c9fa-68ce-4152-ba7a-7440e1274eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc" Oct 30 18:28:56.237557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.237537 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47820c98-v8h9d\" (UID: \"e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d" Oct 30 18:28:56.237631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.237604 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75673761-hlxbc\" (UID: \"a9b2c9fa-68ce-4152-ba7a-7440e1274eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc" Oct 30 18:28:56.238312 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.238284 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-75673761-hlxbc\" (UID: \"a9b2c9fa-68ce-4152-ba7a-7440e1274eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc" Oct 30 18:28:56.238444 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.238284 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-47820c98-v8h9d\" (UID: \"e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d" Oct 30 18:28:56.239803 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.239781 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-maas-api-xks-ca\") pod \"maas-api-revoke-sub-47820c98-v8h9d\" (UID: \"e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d" Oct 30 18:28:56.239930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.239912 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-75673761-hlxbc\" (UID: \"a9b2c9fa-68ce-4152-ba7a-7440e1274eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc" Oct 30 18:28:56.257407 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.257372 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qx2b2\" (UniqueName: \"kubernetes.io/projected/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-kube-api-access-qx2b2\") pod \"maas-api-revoke-sub-47820c98-v8h9d\" (UID: \"e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d" Oct 30 18:28:56.257562 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.257408 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfd46\" (UniqueName: \"kubernetes.io/projected/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-kube-api-access-mfd46\") pod \"maas-api-revoke-sub-75673761-hlxbc\" (UID: \"a9b2c9fa-68ce-4152-ba7a-7440e1274eee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc" Oct 30 18:28:56.346763 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.346728 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc" Oct 30 18:28:56.358022 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.357994 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d" Oct 30 18:28:56.661558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.661527 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc"] Oct 30 18:28:56.661708 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:28:56.661661 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9b2c9fa_68ce_4152_ba7a_7440e1274eee.slice/crio-3925f62b2993d52475f2f95e4b76ff8c93099476a7e0bae5998056ade37fe7b5 WatchSource:0}: Error finding container 3925f62b2993d52475f2f95e4b76ff8c93099476a7e0bae5998056ade37fe7b5: Status 404 returned error can't find the container with id 3925f62b2993d52475f2f95e4b76ff8c93099476a7e0bae5998056ade37fe7b5 Oct 30 18:28:56.670922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.670901 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d"] Oct 30 18:28:56.673145 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:28:56.673120 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode34d5fcc_8eaf_41b1_a11e_5c2ca5edcf49.slice/crio-83c29a9ddf8c8e594a2879d826d5c41952f4d85c166850bdb401f5992858a496 WatchSource:0}: Error finding container 83c29a9ddf8c8e594a2879d826d5c41952f4d85c166850bdb401f5992858a496: Status 404 returned error can't find the container with id 83c29a9ddf8c8e594a2879d826d5c41952f4d85c166850bdb401f5992858a496 Oct 30 18:28:56.841779 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.841742 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc" event={"ID":"a9b2c9fa-68ce-4152-ba7a-7440e1274eee","Type":"ContainerStarted","Data":"3925f62b2993d52475f2f95e4b76ff8c93099476a7e0bae5998056ade37fe7b5"} Oct 30 18:28:56.843243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:56.843218 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d" event={"ID":"e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49","Type":"ContainerStarted","Data":"83c29a9ddf8c8e594a2879d826d5c41952f4d85c166850bdb401f5992858a496"} Oct 30 18:28:57.036422 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:28:57.036389 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:28:57.852704 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:57.852672 5232 generic.go:356] "Generic (PLEG): container finished" podID="e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49" containerID="b751c94e182307ab1a30516d1000d50547d0fb11d9719d4c3f1f571150d86f5a" exitCode=0 Oct 30 18:28:57.852906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:57.852757 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d" event={"ID":"e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49","Type":"ContainerDied","Data":"b751c94e182307ab1a30516d1000d50547d0fb11d9719d4c3f1f571150d86f5a"} Oct 30 18:28:57.854445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:57.854414 5232 generic.go:356] "Generic (PLEG): container finished" podID="a9b2c9fa-68ce-4152-ba7a-7440e1274eee" containerID="f635c709337dde3b11a5a79276e62fa9b4b368edbd455d2266d8b16c94bd143a" exitCode=0 Oct 30 18:28:57.854583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:57.854494 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc" event={"ID":"a9b2c9fa-68ce-4152-ba7a-7440e1274eee","Type":"ContainerDied","Data":"f635c709337dde3b11a5a79276e62fa9b4b368edbd455d2266d8b16c94bd143a"} Oct 30 18:28:59.016618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.016497 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc" Oct 30 18:28:59.062539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.062348 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d" Oct 30 18:28:59.180461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.180370 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-kube-api-access-qx2b2\" (UniqueName: \"kubernetes.io/projected/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-kube-api-access-qx2b2\") pod \"e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49\" (UID: \"e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49\") " Oct 30 18:28:59.180461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.180421 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-kube-api-access-mfd46\" (UniqueName: \"kubernetes.io/projected/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-kube-api-access-mfd46\") pod \"a9b2c9fa-68ce-4152-ba7a-7440e1274eee\" (UID: \"a9b2c9fa-68ce-4152-ba7a-7440e1274eee\") " Oct 30 18:28:59.180739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.180498 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-maas-api-xks-ca\") pod \"e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49\" (UID: \"e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49\") " Oct 30 18:28:59.180739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.180576 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-maas-api-openshift-ca\") pod \"e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49\" (UID: \"e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49\") " Oct 30 18:28:59.180856 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.180792 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-maas-api-xks-ca\") pod \"a9b2c9fa-68ce-4152-ba7a-7440e1274eee\" (UID: \"a9b2c9fa-68ce-4152-ba7a-7440e1274eee\") " Oct 30 18:28:59.180911 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.180896 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-maas-api-openshift-ca\") pod \"a9b2c9fa-68ce-4152-ba7a-7440e1274eee\" (UID: \"a9b2c9fa-68ce-4152-ba7a-7440e1274eee\") " Oct 30 18:28:59.181066 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.181036 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-maas-api-openshift-ca" pod "e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49" (UID: "e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:59.181554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.181529 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:59.181554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.181500 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-maas-api-openshift-ca" pod "a9b2c9fa-68ce-4152-ba7a-7440e1274eee" (UID: "a9b2c9fa-68ce-4152-ba7a-7440e1274eee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:28:59.182976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.182945 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-maas-api-xks-ca" pod "e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49" (UID: "e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:59.183082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.182969 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-kube-api-access-qx2b2" pod "e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49" (UID: "e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49"). InnerVolumeSpecName "kube-api-access-qx2b2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:59.183082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.183014 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-kube-api-access-mfd46" pod "a9b2c9fa-68ce-4152-ba7a-7440e1274eee" (UID: "a9b2c9fa-68ce-4152-ba7a-7440e1274eee"). InnerVolumeSpecName "kube-api-access-mfd46". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:28:59.183184 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.183170 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-maas-api-xks-ca" pod "a9b2c9fa-68ce-4152-ba7a-7440e1274eee" (UID: "a9b2c9fa-68ce-4152-ba7a-7440e1274eee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:28:59.283175 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.283142 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:59.283175 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.283175 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qx2b2\" (UniqueName: \"kubernetes.io/projected/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-kube-api-access-qx2b2\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:59.283175 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.283184 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfd46\" (UniqueName: \"kubernetes.io/projected/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-kube-api-access-mfd46\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:59.283420 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.283194 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:59.283420 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.283202 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a9b2c9fa-68ce-4152-ba7a-7440e1274eee-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:28:59.878789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.877258 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d" event={"ID":"e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49","Type":"ContainerDied","Data":"83c29a9ddf8c8e594a2879d826d5c41952f4d85c166850bdb401f5992858a496"} Oct 30 18:28:59.878789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.877322 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="83c29a9ddf8c8e594a2879d826d5c41952f4d85c166850bdb401f5992858a496" Oct 30 18:28:59.878789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.877408 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d" Oct 30 18:28:59.882816 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.882787 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc" Oct 30 18:28:59.882965 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.882810 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc" event={"ID":"a9b2c9fa-68ce-4152-ba7a-7440e1274eee","Type":"ContainerDied","Data":"3925f62b2993d52475f2f95e4b76ff8c93099476a7e0bae5998056ade37fe7b5"} Oct 30 18:28:59.882965 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:28:59.882877 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3925f62b2993d52475f2f95e4b76ff8c93099476a7e0bae5998056ade37fe7b5" Oct 30 18:29:00.318191 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:00.318154 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg"] Oct 30 18:29:00.323005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:00.322982 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-ef13cc-dx6pg"] Oct 30 18:29:00.803657 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:00.803622 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d151fce2-732c-46f2-9348-0c8889e1f000" path="/var/lib/kubelet/pods/d151fce2-732c-46f2-9348-0c8889e1f000/volumes" Oct 30 18:29:02.070148 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.070114 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w"] Oct 30 18:29:02.070565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.070370 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" podUID="10b2ac99-f96c-4585-81cb-c28db96b7eb6" containerName="istio-proxy" containerID="cri-o://91250cd77bf26f98c8ac4967853884efbf17729bffad7675bae9b90b531c4c88" gracePeriod=30 Oct 30 18:29:02.665451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.665418 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh"] Oct 30 18:29:02.666019 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.666006 5232 cpu_manager.go:414] "containerMap: removing container" podUID="e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49" containerName="revoke-keys" Oct 30 18:29:02.666075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.666020 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49" containerName="revoke-keys" Oct 30 18:29:02.666075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.666041 5232 cpu_manager.go:414] "containerMap: removing container" podUID="a9b2c9fa-68ce-4152-ba7a-7440e1274eee" containerName="revoke-keys" Oct 30 18:29:02.666075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.666046 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9b2c9fa-68ce-4152-ba7a-7440e1274eee" containerName="revoke-keys" Oct 30 18:29:02.666173 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.666141 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49" containerName="revoke-keys" Oct 30 18:29:02.666173 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.666152 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9b2c9fa-68ce-4152-ba7a-7440e1274eee" containerName="revoke-keys" Oct 30 18:29:02.669868 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.669851 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh" Oct 30 18:29:02.689314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.689285 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh"] Oct 30 18:29:02.697027 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.697004 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:29:02.697027 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.697027 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:29:02.745494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.745464 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66256237-e325-4b71-a3d6-78f7e648f661-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh\" (UID: \"66256237-e325-4b71-a3d6-78f7e648f661\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh" Oct 30 18:29:02.745718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.745699 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2wj7\" (UniqueName: \"kubernetes.io/projected/66256237-e325-4b71-a3d6-78f7e648f661-kube-api-access-h2wj7\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh\" (UID: \"66256237-e325-4b71-a3d6-78f7e648f661\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh" Oct 30 18:29:02.745845 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.745829 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66256237-e325-4b71-a3d6-78f7e648f661-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh\" (UID: \"66256237-e325-4b71-a3d6-78f7e648f661\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh" Oct 30 18:29:02.846659 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.846627 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66256237-e325-4b71-a3d6-78f7e648f661-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh\" (UID: \"66256237-e325-4b71-a3d6-78f7e648f661\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh" Oct 30 18:29:02.846866 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.846698 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66256237-e325-4b71-a3d6-78f7e648f661-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh\" (UID: \"66256237-e325-4b71-a3d6-78f7e648f661\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh" Oct 30 18:29:02.846957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.846873 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2wj7\" (UniqueName: \"kubernetes.io/projected/66256237-e325-4b71-a3d6-78f7e648f661-kube-api-access-h2wj7\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh\" (UID: \"66256237-e325-4b71-a3d6-78f7e648f661\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh" Oct 30 18:29:02.847354 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.847327 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66256237-e325-4b71-a3d6-78f7e648f661-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh\" (UID: \"66256237-e325-4b71-a3d6-78f7e648f661\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh" Oct 30 18:29:02.849355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.849335 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66256237-e325-4b71-a3d6-78f7e648f661-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh\" (UID: \"66256237-e325-4b71-a3d6-78f7e648f661\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh" Oct 30 18:29:02.866359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.866333 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2wj7\" (UniqueName: \"kubernetes.io/projected/66256237-e325-4b71-a3d6-78f7e648f661-kube-api-access-h2wj7\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh\" (UID: \"66256237-e325-4b71-a3d6-78f7e648f661\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh" Oct 30 18:29:02.961129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.961041 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" containerID="cri-o://2ad1f5acf196c28ab1eca94aff725423e5403c5269b67ec2d8c9b45a683c6b13" gracePeriod=45 Oct 30 18:29:02.983823 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:02.983800 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh" Oct 30 18:29:03.183170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.183143 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5"] Oct 30 18:29:03.187496 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.187474 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.210011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.209978 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5"] Oct 30 18:29:03.224580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.214996 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0f0c50\"/\"default-dockercfg-swf4t\"" Oct 30 18:29:03.224580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.215054 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0f0c50\"/\"openshift-service-ca.crt\"" Oct 30 18:29:03.224580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.215116 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-0f0c50\"/\"kube-root-ca.crt\"" Oct 30 18:29:03.224580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.215224 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-0f0c50\"/\"test-model-0f0c50-kserve-self-signed-certs\"" Oct 30 18:29:03.252878 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.252832 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-home\") pod \"test-model-0f0c50-kserve-84cdc764c-bjpg5\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.255201 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.255000 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-tls-certs\") pod \"test-model-0f0c50-kserve-84cdc764c-bjpg5\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.255201 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.255149 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2nzc\" (UniqueName: \"kubernetes.io/projected/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-kube-api-access-h2nzc\") pod \"test-model-0f0c50-kserve-84cdc764c-bjpg5\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.255201 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.255197 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-tmp-dir\") pod \"test-model-0f0c50-kserve-84cdc764c-bjpg5\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.255785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.255339 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-dshm\") pod \"test-model-0f0c50-kserve-84cdc764c-bjpg5\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.255785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.255386 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-model-cache\") pod \"test-model-0f0c50-kserve-84cdc764c-bjpg5\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.265975 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.265949 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:29:03.302112 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.302070 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh"] Oct 30 18:29:03.305265 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:29:03.305241 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66256237_e325_4b71_a3d6_78f7e648f661.slice/crio-0d8689d44c61b093a760d3984948c78c0b5c26ce16a00ea957575baa8271760a WatchSource:0}: Error finding container 0d8689d44c61b093a760d3984948c78c0b5c26ce16a00ea957575baa8271760a: Status 404 returned error can't find the container with id 0d8689d44c61b093a760d3984948c78c0b5c26ce16a00ea957575baa8271760a Oct 30 18:29:03.357777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.356133 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7634ee23-ee09-4bd9-92a1-4a56b4a82460-tls-certs\" (UniqueName: \"kubernetes.io/secret/7634ee23-ee09-4bd9-92a1-4a56b4a82460-tls-certs\") pod \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " Oct 30 18:29:03.357777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.356299 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7634ee23-ee09-4bd9-92a1-4a56b4a82460-kube-api-access-bfpnt\" (UniqueName: \"kubernetes.io/projected/7634ee23-ee09-4bd9-92a1-4a56b4a82460-kube-api-access-bfpnt\") pod \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " Oct 30 18:29:03.357777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.356332 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-model-cache\") pod \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " Oct 30 18:29:03.357777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.356373 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-home\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-home\") pod \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " Oct 30 18:29:03.357777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.356402 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-tmp-dir\") pod \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " Oct 30 18:29:03.357777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.356434 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-dshm\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-dshm\") pod \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\" (UID: \"7634ee23-ee09-4bd9-92a1-4a56b4a82460\") " Oct 30 18:29:03.357777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.356831 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-tmp-dir\") pod \"test-model-0f0c50-kserve-84cdc764c-bjpg5\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.357777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.356991 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-dshm\") pod \"test-model-0f0c50-kserve-84cdc764c-bjpg5\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.357777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.357033 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-model-cache\") pod \"test-model-0f0c50-kserve-84cdc764c-bjpg5\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.357777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.357142 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-home\") pod \"test-model-0f0c50-kserve-84cdc764c-bjpg5\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.357777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.357187 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-tls-certs\") pod \"test-model-0f0c50-kserve-84cdc764c-bjpg5\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.357777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.357304 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2nzc\" (UniqueName: \"kubernetes.io/projected/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-kube-api-access-h2nzc\") pod \"test-model-0f0c50-kserve-84cdc764c-bjpg5\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.358853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.358480 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-model-cache\") pod \"test-model-0f0c50-kserve-84cdc764c-bjpg5\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.358928 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.358848 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-tmp-dir\") pod \"test-model-0f0c50-kserve-84cdc764c-bjpg5\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.359670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.359641 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-home\") pod \"test-model-0f0c50-kserve-84cdc764c-bjpg5\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.360136 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.359784 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-model-cache" pod "7634ee23-ee09-4bd9-92a1-4a56b4a82460" (UID: "7634ee23-ee09-4bd9-92a1-4a56b4a82460"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:03.360423 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.360390 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-home" pod "7634ee23-ee09-4bd9-92a1-4a56b4a82460" (UID: "7634ee23-ee09-4bd9-92a1-4a56b4a82460"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:03.360617 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.360592 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-tmp-dir" pod "7634ee23-ee09-4bd9-92a1-4a56b4a82460" (UID: "7634ee23-ee09-4bd9-92a1-4a56b4a82460"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:03.361307 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.361281 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7634ee23-ee09-4bd9-92a1-4a56b4a82460-tls-certs" pod "7634ee23-ee09-4bd9-92a1-4a56b4a82460" (UID: "7634ee23-ee09-4bd9-92a1-4a56b4a82460"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:03.361803 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.361778 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-dshm\") pod \"test-model-0f0c50-kserve-84cdc764c-bjpg5\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.362802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.362776 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7634ee23-ee09-4bd9-92a1-4a56b4a82460-kube-api-access-bfpnt" pod "7634ee23-ee09-4bd9-92a1-4a56b4a82460" (UID: "7634ee23-ee09-4bd9-92a1-4a56b4a82460"). InnerVolumeSpecName "kube-api-access-bfpnt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:03.362921 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.362900 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-dshm" pod "7634ee23-ee09-4bd9-92a1-4a56b4a82460" (UID: "7634ee23-ee09-4bd9-92a1-4a56b4a82460"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:03.364228 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.364203 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-tls-certs\") pod \"test-model-0f0c50-kserve-84cdc764c-bjpg5\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.377171 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.377141 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2nzc\" (UniqueName: \"kubernetes.io/projected/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-kube-api-access-h2nzc\") pod \"test-model-0f0c50-kserve-84cdc764c-bjpg5\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.458934 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.458901 5232 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7634ee23-ee09-4bd9-92a1-4a56b4a82460-tls-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.458934 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.458931 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bfpnt\" (UniqueName: \"kubernetes.io/projected/7634ee23-ee09-4bd9-92a1-4a56b4a82460-kube-api-access-bfpnt\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.459146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.458944 5232 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-model-cache\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.459146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.458955 5232 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-home\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.459146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.458965 5232 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-tmp-dir\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.459146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.458975 5232 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7634ee23-ee09-4bd9-92a1-4a56b4a82460-dshm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:03.523931 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.523900 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:03.913321 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.913288 5232 generic.go:356] "Generic (PLEG): container finished" podID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerID="2ad1f5acf196c28ab1eca94aff725423e5403c5269b67ec2d8c9b45a683c6b13" exitCode=0 Oct 30 18:29:03.913503 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.913361 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" Oct 30 18:29:03.913595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.913499 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" event={"ID":"7634ee23-ee09-4bd9-92a1-4a56b4a82460","Type":"ContainerDied","Data":"2ad1f5acf196c28ab1eca94aff725423e5403c5269b67ec2d8c9b45a683c6b13"} Oct 30 18:29:03.913595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.913568 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" event={"ID":"7634ee23-ee09-4bd9-92a1-4a56b4a82460","Type":"ContainerDied","Data":"907c8b95e592c711b91b7ea82dc32cb571c657b0c2508a65617703e41b33d365"} Oct 30 18:29:03.913710 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.913594 5232 scope.go:122] "RemoveContainer" containerID="2ad1f5acf196c28ab1eca94aff725423e5403c5269b67ec2d8c9b45a683c6b13" Oct 30 18:29:03.920231 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.920200 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh" event={"ID":"66256237-e325-4b71-a3d6-78f7e648f661","Type":"ContainerStarted","Data":"463045d630af32830791c9940546a65573f3204f5e1d29ec4934fbf40f3863ec"} Oct 30 18:29:03.920342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.920259 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh" event={"ID":"66256237-e325-4b71-a3d6-78f7e648f661","Type":"ContainerStarted","Data":"0d8689d44c61b093a760d3984948c78c0b5c26ce16a00ea957575baa8271760a"} Oct 30 18:29:03.930694 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.930669 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5"] Oct 30 18:29:03.931478 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:29:03.931455 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0dba5d19_7ec3_4fdc_ba5f_4e9c695d70c9.slice/crio-58f46d0a9183df387d38bb821333d09c88eedc8ac355a3a5e453b6dad18209d4 WatchSource:0}: Error finding container 58f46d0a9183df387d38bb821333d09c88eedc8ac355a3a5e453b6dad18209d4: Status 404 returned error can't find the container with id 58f46d0a9183df387d38bb821333d09c88eedc8ac355a3a5e453b6dad18209d4 Oct 30 18:29:03.936894 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.936870 5232 scope.go:122] "RemoveContainer" containerID="2ad1f5acf196c28ab1eca94aff725423e5403c5269b67ec2d8c9b45a683c6b13" Oct 30 18:29:03.937181 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:29:03.937162 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ad1f5acf196c28ab1eca94aff725423e5403c5269b67ec2d8c9b45a683c6b13\": container with ID starting with 2ad1f5acf196c28ab1eca94aff725423e5403c5269b67ec2d8c9b45a683c6b13 not found: ID does not exist" containerID="2ad1f5acf196c28ab1eca94aff725423e5403c5269b67ec2d8c9b45a683c6b13" Oct 30 18:29:03.937255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.937189 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ad1f5acf196c28ab1eca94aff725423e5403c5269b67ec2d8c9b45a683c6b13"} err="failed to get container status \"2ad1f5acf196c28ab1eca94aff725423e5403c5269b67ec2d8c9b45a683c6b13\": rpc error: code = NotFound desc = could not find container \"2ad1f5acf196c28ab1eca94aff725423e5403c5269b67ec2d8c9b45a683c6b13\": container with ID starting with 2ad1f5acf196c28ab1eca94aff725423e5403c5269b67ec2d8c9b45a683c6b13 not found: ID does not exist" Oct 30 18:29:03.966023 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.966004 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t"] Oct 30 18:29:03.968941 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:03.968920 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t"] Oct 30 18:29:04.007825 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:04.007774 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh" podStartSLOduration=2.007761371 podStartE2EDuration="2.007761371s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:04.00563923 +0000 UTC m=+1643.322059552" watchObservedRunningTime="2026-10-30 18:29:04.007761371 +0000 UTC m=+1643.324181695" Oct 30 18:29:04.197817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:04.197732 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-6ce5eb/simulator-w0-6ce5eb-kserve-558b9cf76f-ws58t" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" probeResult="failure" output="Get \"https://10.133.0.46:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 30 18:29:04.803490 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:04.803460 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" path="/var/lib/kubelet/pods/7634ee23-ee09-4bd9-92a1-4a56b4a82460/volumes" Oct 30 18:29:04.929540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:04.929486 5232 generic.go:356] "Generic (PLEG): container finished" podID="66256237-e325-4b71-a3d6-78f7e648f661" containerID="463045d630af32830791c9940546a65573f3204f5e1d29ec4934fbf40f3863ec" exitCode=0 Oct 30 18:29:04.929733 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:04.929540 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh" event={"ID":"66256237-e325-4b71-a3d6-78f7e648f661","Type":"ContainerDied","Data":"463045d630af32830791c9940546a65573f3204f5e1d29ec4934fbf40f3863ec"} Oct 30 18:29:04.931387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:04.931366 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" event={"ID":"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9","Type":"ContainerStarted","Data":"eccc08df0cc87f67fabc4d777c773dfb180f1e6d83c26e73238ad44c76abe095"} Oct 30 18:29:04.931520 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:04.931397 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" event={"ID":"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9","Type":"ContainerStarted","Data":"58f46d0a9183df387d38bb821333d09c88eedc8ac355a3a5e453b6dad18209d4"} Oct 30 18:29:05.014391 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:05.014331 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" podStartSLOduration=2.014310283 podStartE2EDuration="2.014310283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:05.010570188 +0000 UTC m=+1644.326990510" watchObservedRunningTime="2026-10-30 18:29:05.014310283 +0000 UTC m=+1644.330730614" Oct 30 18:29:05.240911 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:05.240880 5232 patch_prober.go:29] interesting pod/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:29:05.241295 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:05.240928 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" podUID="10b2ac99-f96c-4585-81cb-c28db96b7eb6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:29:05.524048 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:05.523971 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:06.199654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:06.198656 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh" Oct 30 18:29:06.306775 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:06.306747 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/66256237-e325-4b71-a3d6-78f7e648f661-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66256237-e325-4b71-a3d6-78f7e648f661-maas-api-openshift-ca\") pod \"66256237-e325-4b71-a3d6-78f7e648f661\" (UID: \"66256237-e325-4b71-a3d6-78f7e648f661\") " Oct 30 18:29:06.307216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:06.306834 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/66256237-e325-4b71-a3d6-78f7e648f661-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66256237-e325-4b71-a3d6-78f7e648f661-maas-api-xks-ca\") pod \"66256237-e325-4b71-a3d6-78f7e648f661\" (UID: \"66256237-e325-4b71-a3d6-78f7e648f661\") " Oct 30 18:29:06.307216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:06.307001 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/66256237-e325-4b71-a3d6-78f7e648f661-kube-api-access-h2wj7\" (UniqueName: \"kubernetes.io/projected/66256237-e325-4b71-a3d6-78f7e648f661-kube-api-access-h2wj7\") pod \"66256237-e325-4b71-a3d6-78f7e648f661\" (UID: \"66256237-e325-4b71-a3d6-78f7e648f661\") " Oct 30 18:29:06.307216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:06.307064 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66256237-e325-4b71-a3d6-78f7e648f661-maas-api-openshift-ca" pod "66256237-e325-4b71-a3d6-78f7e648f661" (UID: "66256237-e325-4b71-a3d6-78f7e648f661"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:06.307452 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:06.307433 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/66256237-e325-4b71-a3d6-78f7e648f661-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:06.309013 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:06.308986 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/66256237-e325-4b71-a3d6-78f7e648f661-maas-api-xks-ca" pod "66256237-e325-4b71-a3d6-78f7e648f661" (UID: "66256237-e325-4b71-a3d6-78f7e648f661"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:06.309163 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:06.309137 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66256237-e325-4b71-a3d6-78f7e648f661-kube-api-access-h2wj7" pod "66256237-e325-4b71-a3d6-78f7e648f661" (UID: "66256237-e325-4b71-a3d6-78f7e648f661"). InnerVolumeSpecName "kube-api-access-h2wj7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:06.408921 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:06.408890 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2wj7\" (UniqueName: \"kubernetes.io/projected/66256237-e325-4b71-a3d6-78f7e648f661-kube-api-access-h2wj7\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:06.408921 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:06.408918 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/66256237-e325-4b71-a3d6-78f7e648f661-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:06.948434 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:06.948404 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh" Oct 30 18:29:06.948620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:06.948442 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh" event={"ID":"66256237-e325-4b71-a3d6-78f7e648f661","Type":"ContainerDied","Data":"0d8689d44c61b093a760d3984948c78c0b5c26ce16a00ea957575baa8271760a"} Oct 30 18:29:06.948620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:06.948486 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d8689d44c61b093a760d3984948c78c0b5c26ce16a00ea957575baa8271760a" Oct 30 18:29:07.338729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.338672 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:29:07.421841 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.421812 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-credential-socket\") pod \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " Oct 30 18:29:07.422015 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.421926 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-workload-socket\") pod \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " Oct 30 18:29:07.422015 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.421984 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-workload-certs\") pod \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " Oct 30 18:29:07.422015 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.422005 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istiod-ca-cert\") pod \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " Oct 30 18:29:07.422194 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.422058 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-podinfo\") pod \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " Oct 30 18:29:07.422194 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.422096 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-data\") pod \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " Oct 30 18:29:07.422194 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.422146 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-credential-socket" pod "10b2ac99-f96c-4585-81cb-c28db96b7eb6" (UID: "10b2ac99-f96c-4585-81cb-c28db96b7eb6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:07.422194 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.422160 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-envoy\") pod \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " Oct 30 18:29:07.422426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.422200 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-token\" (UniqueName: \"kubernetes.io/projected/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-token\") pod \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " Oct 30 18:29:07.422426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.422273 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10b2ac99-f96c-4585-81cb-c28db96b7eb6-kube-api-access-l2vnd\" (UniqueName: \"kubernetes.io/projected/10b2ac99-f96c-4585-81cb-c28db96b7eb6-kube-api-access-l2vnd\") pod \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\" (UID: \"10b2ac99-f96c-4585-81cb-c28db96b7eb6\") " Oct 30 18:29:07.422426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.422294 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-workload-socket" pod "10b2ac99-f96c-4585-81cb-c28db96b7eb6" (UID: "10b2ac99-f96c-4585-81cb-c28db96b7eb6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:07.422926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.422775 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-data" pod "10b2ac99-f96c-4585-81cb-c28db96b7eb6" (UID: "10b2ac99-f96c-4585-81cb-c28db96b7eb6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:07.423103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.422923 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.423103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.422947 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.423103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.422962 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.423103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.423065 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istiod-ca-cert" pod "10b2ac99-f96c-4585-81cb-c28db96b7eb6" (UID: "10b2ac99-f96c-4585-81cb-c28db96b7eb6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:07.425035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.425003 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-workload-certs" pod "10b2ac99-f96c-4585-81cb-c28db96b7eb6" (UID: "10b2ac99-f96c-4585-81cb-c28db96b7eb6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:07.425942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.425918 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-token" pod "10b2ac99-f96c-4585-81cb-c28db96b7eb6" (UID: "10b2ac99-f96c-4585-81cb-c28db96b7eb6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:07.426050 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.425952 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10b2ac99-f96c-4585-81cb-c28db96b7eb6-kube-api-access-l2vnd" pod "10b2ac99-f96c-4585-81cb-c28db96b7eb6" (UID: "10b2ac99-f96c-4585-81cb-c28db96b7eb6"). InnerVolumeSpecName "kube-api-access-l2vnd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:07.426050 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.425991 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-envoy" pod "10b2ac99-f96c-4585-81cb-c28db96b7eb6" (UID: "10b2ac99-f96c-4585-81cb-c28db96b7eb6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:07.426189 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.426069 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-podinfo" pod "10b2ac99-f96c-4585-81cb-c28db96b7eb6" (UID: "10b2ac99-f96c-4585-81cb-c28db96b7eb6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:29:07.547128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.531020 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.547128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.531058 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.547128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.531072 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.547128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.531085 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.547128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.531097 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/10b2ac99-f96c-4585-81cb-c28db96b7eb6-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.547128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.531110 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2vnd\" (UniqueName: \"kubernetes.io/projected/10b2ac99-f96c-4585-81cb-c28db96b7eb6-kube-api-access-l2vnd\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:07.990543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.989161 5232 generic.go:356] "Generic (PLEG): container finished" podID="10b2ac99-f96c-4585-81cb-c28db96b7eb6" containerID="91250cd77bf26f98c8ac4967853884efbf17729bffad7675bae9b90b531c4c88" exitCode=0 Oct 30 18:29:07.990543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.989319 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" event={"ID":"10b2ac99-f96c-4585-81cb-c28db96b7eb6","Type":"ContainerDied","Data":"91250cd77bf26f98c8ac4967853884efbf17729bffad7675bae9b90b531c4c88"} Oct 30 18:29:07.990543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.989370 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" event={"ID":"10b2ac99-f96c-4585-81cb-c28db96b7eb6","Type":"ContainerDied","Data":"5276ab1dab391f2b6902d1a70c13049ea9b48530a504a3029156ed01d1759d31"} Oct 30 18:29:07.990543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.989398 5232 scope.go:122] "RemoveContainer" containerID="91250cd77bf26f98c8ac4967853884efbf17729bffad7675bae9b90b531c4c88" Oct 30 18:29:07.990543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:07.989597 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w" Oct 30 18:29:08.044842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:08.044063 5232 scope.go:122] "RemoveContainer" containerID="91250cd77bf26f98c8ac4967853884efbf17729bffad7675bae9b90b531c4c88" Oct 30 18:29:08.045861 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:29:08.045760 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"91250cd77bf26f98c8ac4967853884efbf17729bffad7675bae9b90b531c4c88\": container with ID starting with 91250cd77bf26f98c8ac4967853884efbf17729bffad7675bae9b90b531c4c88 not found: ID does not exist" containerID="91250cd77bf26f98c8ac4967853884efbf17729bffad7675bae9b90b531c4c88" Oct 30 18:29:08.045861 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:08.045797 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"91250cd77bf26f98c8ac4967853884efbf17729bffad7675bae9b90b531c4c88"} err="failed to get container status \"91250cd77bf26f98c8ac4967853884efbf17729bffad7675bae9b90b531c4c88\": rpc error: code = NotFound desc = could not find container \"91250cd77bf26f98c8ac4967853884efbf17729bffad7675bae9b90b531c4c88\": container with ID starting with 91250cd77bf26f98c8ac4967853884efbf17729bffad7675bae9b90b531c4c88 not found: ID does not exist" Oct 30 18:29:08.052024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:08.051963 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m"] Oct 30 18:29:08.061629 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:08.061357 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-36a4e9a2-x845m"] Oct 30 18:29:08.071891 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:08.071829 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w"] Oct 30 18:29:08.075406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:08.075380 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-b2b596-openshift-default-7968fb6f7-bkm9w"] Oct 30 18:29:08.802916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:08.802881 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10b2ac99-f96c-4585-81cb-c28db96b7eb6" path="/var/lib/kubelet/pods/10b2ac99-f96c-4585-81cb-c28db96b7eb6/volumes" Oct 30 18:29:08.803326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:08.803311 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cbe19226-83bf-4dc4-883d-e24502a99599" path="/var/lib/kubelet/pods/cbe19226-83bf-4dc4-883d-e24502a99599/volumes" Oct 30 18:29:13.524624 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:13.524592 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:13.538018 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:13.537993 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:14.168611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:14.168574 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:29:16.551498 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:16.551455 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs"] Oct 30 18:29:16.564146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:16.564114 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-6ce5eb-6b7xs"] Oct 30 18:29:16.813302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:16.813222 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4f1cecd6-c961-445c-955e-5a8f8b4f3168" path="/var/lib/kubelet/pods/4f1cecd6-c961-445c-955e-5a8f8b4f3168/volumes" Oct 30 18:29:24.323823 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.323781 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk"] Oct 30 18:29:24.324484 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.324465 5232 cpu_manager.go:414] "containerMap: removing container" podUID="66256237-e325-4b71-a3d6-78f7e648f661" containerName="revoke-keys" Oct 30 18:29:24.324560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.324488 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="66256237-e325-4b71-a3d6-78f7e648f661" containerName="revoke-keys" Oct 30 18:29:24.324560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.324550 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" Oct 30 18:29:24.324560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.324559 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" Oct 30 18:29:24.324680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.324614 5232 cpu_manager.go:414] "containerMap: removing container" podUID="10b2ac99-f96c-4585-81cb-c28db96b7eb6" containerName="istio-proxy" Oct 30 18:29:24.324680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.324620 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10b2ac99-f96c-4585-81cb-c28db96b7eb6" containerName="istio-proxy" Oct 30 18:29:24.324760 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.324735 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7634ee23-ee09-4bd9-92a1-4a56b4a82460" containerName="main" Oct 30 18:29:24.324760 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.324750 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="10b2ac99-f96c-4585-81cb-c28db96b7eb6" containerName="istio-proxy" Oct 30 18:29:24.324760 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.324757 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="66256237-e325-4b71-a3d6-78f7e648f661" containerName="revoke-keys" Oct 30 18:29:24.329652 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.329633 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk" Oct 30 18:29:24.347115 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.347089 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk"] Oct 30 18:29:24.354808 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.354781 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:29:24.355000 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.354860 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:29:24.416850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.416812 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482da54b-63ab-4595-9d44-3ca24ed32d03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk\" (UID: \"482da54b-63ab-4595-9d44-3ca24ed32d03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk" Oct 30 18:29:24.417237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.417193 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482da54b-63ab-4595-9d44-3ca24ed32d03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk\" (UID: \"482da54b-63ab-4595-9d44-3ca24ed32d03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk" Oct 30 18:29:24.417349 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.417290 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j7lxz\" (UniqueName: \"kubernetes.io/projected/482da54b-63ab-4595-9d44-3ca24ed32d03-kube-api-access-j7lxz\") pod \"maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk\" (UID: \"482da54b-63ab-4595-9d44-3ca24ed32d03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk" Oct 30 18:29:24.518689 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.518642 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482da54b-63ab-4595-9d44-3ca24ed32d03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk\" (UID: \"482da54b-63ab-4595-9d44-3ca24ed32d03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk" Oct 30 18:29:24.518884 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.518748 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482da54b-63ab-4595-9d44-3ca24ed32d03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk\" (UID: \"482da54b-63ab-4595-9d44-3ca24ed32d03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk" Oct 30 18:29:24.518884 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.518796 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j7lxz\" (UniqueName: \"kubernetes.io/projected/482da54b-63ab-4595-9d44-3ca24ed32d03-kube-api-access-j7lxz\") pod \"maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk\" (UID: \"482da54b-63ab-4595-9d44-3ca24ed32d03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk" Oct 30 18:29:24.519388 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.519364 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482da54b-63ab-4595-9d44-3ca24ed32d03-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk\" (UID: \"482da54b-63ab-4595-9d44-3ca24ed32d03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk" Oct 30 18:29:24.521399 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.521377 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482da54b-63ab-4595-9d44-3ca24ed32d03-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk\" (UID: \"482da54b-63ab-4595-9d44-3ca24ed32d03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk" Oct 30 18:29:24.554719 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.554681 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j7lxz\" (UniqueName: \"kubernetes.io/projected/482da54b-63ab-4595-9d44-3ca24ed32d03-kube-api-access-j7lxz\") pod \"maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk\" (UID: \"482da54b-63ab-4595-9d44-3ca24ed32d03\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk" Oct 30 18:29:24.645200 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.645105 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk" Oct 30 18:29:24.977075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:24.977047 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk"] Oct 30 18:29:24.980586 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:29:24.980551 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod482da54b_63ab_4595_9d44_3ca24ed32d03.slice/crio-70ed04a219a2df41c2f47694186edc35a7e1d6e556e162be382238764937406d WatchSource:0}: Error finding container 70ed04a219a2df41c2f47694186edc35a7e1d6e556e162be382238764937406d: Status 404 returned error can't find the container with id 70ed04a219a2df41c2f47694186edc35a7e1d6e556e162be382238764937406d Oct 30 18:29:25.232781 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:25.232689 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk" event={"ID":"482da54b-63ab-4595-9d44-3ca24ed32d03","Type":"ContainerStarted","Data":"70ed04a219a2df41c2f47694186edc35a7e1d6e556e162be382238764937406d"} Oct 30 18:29:26.240968 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:26.240932 5232 generic.go:356] "Generic (PLEG): container finished" podID="482da54b-63ab-4595-9d44-3ca24ed32d03" containerID="ed1575ed85c1db9547135d18971d4920b6fff46ef29ccfbf04c823180eae59da" exitCode=0 Oct 30 18:29:26.241412 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:26.241030 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk" event={"ID":"482da54b-63ab-4595-9d44-3ca24ed32d03","Type":"ContainerDied","Data":"ed1575ed85c1db9547135d18971d4920b6fff46ef29ccfbf04c823180eae59da"} Oct 30 18:29:26.546191 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:26.546154 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m"] Oct 30 18:29:26.546421 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:26.546392 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" podUID="cc39faea-f977-4ba2-be13-cef341667507" containerName="istio-proxy" containerID="cri-o://d561c4c444e09fb77515150263580a4cca237d0bdc7fb8ee2aad28e974ce4aa8" gracePeriod=30 Oct 30 18:29:27.060727 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:29:27.060687 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:29:27.529359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:27.528923 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk" Oct 30 18:29:27.599457 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:27.599354 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/482da54b-63ab-4595-9d44-3ca24ed32d03-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482da54b-63ab-4595-9d44-3ca24ed32d03-maas-api-openshift-ca\") pod \"482da54b-63ab-4595-9d44-3ca24ed32d03\" (UID: \"482da54b-63ab-4595-9d44-3ca24ed32d03\") " Oct 30 18:29:27.599706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:27.599574 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/482da54b-63ab-4595-9d44-3ca24ed32d03-kube-api-access-j7lxz\" (UniqueName: \"kubernetes.io/projected/482da54b-63ab-4595-9d44-3ca24ed32d03-kube-api-access-j7lxz\") pod \"482da54b-63ab-4595-9d44-3ca24ed32d03\" (UID: \"482da54b-63ab-4595-9d44-3ca24ed32d03\") " Oct 30 18:29:27.599706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:27.599633 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/482da54b-63ab-4595-9d44-3ca24ed32d03-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482da54b-63ab-4595-9d44-3ca24ed32d03-maas-api-xks-ca\") pod \"482da54b-63ab-4595-9d44-3ca24ed32d03\" (UID: \"482da54b-63ab-4595-9d44-3ca24ed32d03\") " Oct 30 18:29:27.601938 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:27.601874 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/482da54b-63ab-4595-9d44-3ca24ed32d03-maas-api-openshift-ca" pod "482da54b-63ab-4595-9d44-3ca24ed32d03" (UID: "482da54b-63ab-4595-9d44-3ca24ed32d03"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:27.603281 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:27.603250 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/482da54b-63ab-4595-9d44-3ca24ed32d03-maas-api-xks-ca" pod "482da54b-63ab-4595-9d44-3ca24ed32d03" (UID: "482da54b-63ab-4595-9d44-3ca24ed32d03"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:27.603640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:27.603616 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/482da54b-63ab-4595-9d44-3ca24ed32d03-kube-api-access-j7lxz" pod "482da54b-63ab-4595-9d44-3ca24ed32d03" (UID: "482da54b-63ab-4595-9d44-3ca24ed32d03"). InnerVolumeSpecName "kube-api-access-j7lxz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:27.720573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:27.707589 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j7lxz\" (UniqueName: \"kubernetes.io/projected/482da54b-63ab-4595-9d44-3ca24ed32d03-kube-api-access-j7lxz\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:27.720573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:27.707641 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/482da54b-63ab-4595-9d44-3ca24ed32d03-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:27.720573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:27.707656 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/482da54b-63ab-4595-9d44-3ca24ed32d03-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:28.064190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:28.064154 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh"] Oct 30 18:29:28.067463 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:28.067430 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-273105-wl4zh"] Oct 30 18:29:28.285670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:28.285634 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk" Oct 30 18:29:28.286656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:28.285701 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk" event={"ID":"482da54b-63ab-4595-9d44-3ca24ed32d03","Type":"ContainerDied","Data":"70ed04a219a2df41c2f47694186edc35a7e1d6e556e162be382238764937406d"} Oct 30 18:29:28.286893 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:28.286872 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="70ed04a219a2df41c2f47694186edc35a7e1d6e556e162be382238764937406d" Oct 30 18:29:28.804189 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:28.804151 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="66256237-e325-4b71-a3d6-78f7e648f661" path="/var/lib/kubelet/pods/66256237-e325-4b71-a3d6-78f7e648f661/volumes" Oct 30 18:29:29.150041 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:29.149949 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd"] Oct 30 18:29:29.179886 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:29.179855 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w"] Oct 30 18:29:31.866929 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:31.866898 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:29:31.999687 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:31.999646 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc39faea-f977-4ba2-be13-cef341667507-istio-token\" (UniqueName: \"kubernetes.io/projected/cc39faea-f977-4ba2-be13-cef341667507-istio-token\") pod \"cc39faea-f977-4ba2-be13-cef341667507\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " Oct 30 18:29:31.999886 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:31.999726 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-workload-socket\") pod \"cc39faea-f977-4ba2-be13-cef341667507\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " Oct 30 18:29:31.999886 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:31.999754 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc39faea-f977-4ba2-be13-cef341667507-kube-api-access-65qkg\" (UniqueName: \"kubernetes.io/projected/cc39faea-f977-4ba2-be13-cef341667507-kube-api-access-65qkg\") pod \"cc39faea-f977-4ba2-be13-cef341667507\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " Oct 30 18:29:31.999886 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:31.999816 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cc39faea-f977-4ba2-be13-cef341667507-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc39faea-f977-4ba2-be13-cef341667507-istio-podinfo\") pod \"cc39faea-f977-4ba2-be13-cef341667507\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " Oct 30 18:29:31.999886 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:31.999849 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-credential-socket\") pod \"cc39faea-f977-4ba2-be13-cef341667507\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " Oct 30 18:29:32.000118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:31.999903 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-istio-envoy\") pod \"cc39faea-f977-4ba2-be13-cef341667507\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " Oct 30 18:29:32.000118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:31.999955 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-workload-certs\") pod \"cc39faea-f977-4ba2-be13-cef341667507\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " Oct 30 18:29:32.000118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.000008 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc39faea-f977-4ba2-be13-cef341667507-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc39faea-f977-4ba2-be13-cef341667507-istiod-ca-cert\") pod \"cc39faea-f977-4ba2-be13-cef341667507\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " Oct 30 18:29:32.000118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.000041 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-istio-data\") pod \"cc39faea-f977-4ba2-be13-cef341667507\" (UID: \"cc39faea-f977-4ba2-be13-cef341667507\") " Oct 30 18:29:32.001207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.000779 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-istio-data" pod "cc39faea-f977-4ba2-be13-cef341667507" (UID: "cc39faea-f977-4ba2-be13-cef341667507"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:32.001348 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.001266 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-workload-certs" pod "cc39faea-f977-4ba2-be13-cef341667507" (UID: "cc39faea-f977-4ba2-be13-cef341667507"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:32.001348 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.001278 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-credential-socket" pod "cc39faea-f977-4ba2-be13-cef341667507" (UID: "cc39faea-f977-4ba2-be13-cef341667507"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:32.001348 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.001283 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-workload-socket" pod "cc39faea-f977-4ba2-be13-cef341667507" (UID: "cc39faea-f977-4ba2-be13-cef341667507"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:32.001474 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.001348 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc39faea-f977-4ba2-be13-cef341667507-istiod-ca-cert" pod "cc39faea-f977-4ba2-be13-cef341667507" (UID: "cc39faea-f977-4ba2-be13-cef341667507"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:32.003254 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.003230 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc39faea-f977-4ba2-be13-cef341667507-istio-token" pod "cc39faea-f977-4ba2-be13-cef341667507" (UID: "cc39faea-f977-4ba2-be13-cef341667507"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:32.003349 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.003284 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc39faea-f977-4ba2-be13-cef341667507-kube-api-access-65qkg" pod "cc39faea-f977-4ba2-be13-cef341667507" (UID: "cc39faea-f977-4ba2-be13-cef341667507"). InnerVolumeSpecName "kube-api-access-65qkg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:32.003541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.003523 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-istio-envoy" pod "cc39faea-f977-4ba2-be13-cef341667507" (UID: "cc39faea-f977-4ba2-be13-cef341667507"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:32.003664 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.003641 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cc39faea-f977-4ba2-be13-cef341667507-istio-podinfo" pod "cc39faea-f977-4ba2-be13-cef341667507" (UID: "cc39faea-f977-4ba2-be13-cef341667507"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:29:32.107293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.107215 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:32.107293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.107244 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-65qkg\" (UniqueName: \"kubernetes.io/projected/cc39faea-f977-4ba2-be13-cef341667507-kube-api-access-65qkg\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:32.107293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.107254 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cc39faea-f977-4ba2-be13-cef341667507-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:32.107293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.107263 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:32.107293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.107272 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:32.107293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.107281 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:32.107293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.107288 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cc39faea-f977-4ba2-be13-cef341667507-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:32.107293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.107295 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cc39faea-f977-4ba2-be13-cef341667507-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:32.107293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.107302 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cc39faea-f977-4ba2-be13-cef341667507-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:32.341392 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.341353 5232 generic.go:356] "Generic (PLEG): container finished" podID="cc39faea-f977-4ba2-be13-cef341667507" containerID="d561c4c444e09fb77515150263580a4cca237d0bdc7fb8ee2aad28e974ce4aa8" exitCode=0 Oct 30 18:29:32.341623 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.341397 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" Oct 30 18:29:32.341623 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.341431 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" event={"ID":"cc39faea-f977-4ba2-be13-cef341667507","Type":"ContainerDied","Data":"d561c4c444e09fb77515150263580a4cca237d0bdc7fb8ee2aad28e974ce4aa8"} Oct 30 18:29:32.341623 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.341495 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m" event={"ID":"cc39faea-f977-4ba2-be13-cef341667507","Type":"ContainerDied","Data":"8bfc688071418547f9692663c15fc0a6158b7472d07fc7e8ee971a705205ed9f"} Oct 30 18:29:32.341623 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.341546 5232 scope.go:122] "RemoveContainer" containerID="d561c4c444e09fb77515150263580a4cca237d0bdc7fb8ee2aad28e974ce4aa8" Oct 30 18:29:32.355052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.355026 5232 scope.go:122] "RemoveContainer" containerID="d561c4c444e09fb77515150263580a4cca237d0bdc7fb8ee2aad28e974ce4aa8" Oct 30 18:29:32.355319 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:29:32.355302 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d561c4c444e09fb77515150263580a4cca237d0bdc7fb8ee2aad28e974ce4aa8\": container with ID starting with d561c4c444e09fb77515150263580a4cca237d0bdc7fb8ee2aad28e974ce4aa8 not found: ID does not exist" containerID="d561c4c444e09fb77515150263580a4cca237d0bdc7fb8ee2aad28e974ce4aa8" Oct 30 18:29:32.355368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.355331 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d561c4c444e09fb77515150263580a4cca237d0bdc7fb8ee2aad28e974ce4aa8"} err="failed to get container status \"d561c4c444e09fb77515150263580a4cca237d0bdc7fb8ee2aad28e974ce4aa8\": rpc error: code = NotFound desc = could not find container \"d561c4c444e09fb77515150263580a4cca237d0bdc7fb8ee2aad28e974ce4aa8\": container with ID starting with d561c4c444e09fb77515150263580a4cca237d0bdc7fb8ee2aad28e974ce4aa8 not found: ID does not exist" Oct 30 18:29:32.393775 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.393695 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m"] Oct 30 18:29:32.406273 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.406241 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-r4t5m"] Oct 30 18:29:32.803047 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:32.803019 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc39faea-f977-4ba2-be13-cef341667507" path="/var/lib/kubelet/pods/cc39faea-f977-4ba2-be13-cef341667507/volumes" Oct 30 18:29:34.813339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:34.813302 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d"] Oct 30 18:29:34.846760 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:34.846712 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj"] Oct 30 18:29:36.020699 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.020660 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f"] Oct 30 18:29:36.021662 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.021640 5232 cpu_manager.go:414] "containerMap: removing container" podUID="cc39faea-f977-4ba2-be13-cef341667507" containerName="istio-proxy" Oct 30 18:29:36.021779 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.021664 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc39faea-f977-4ba2-be13-cef341667507" containerName="istio-proxy" Oct 30 18:29:36.021779 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.021719 5232 cpu_manager.go:414] "containerMap: removing container" podUID="482da54b-63ab-4595-9d44-3ca24ed32d03" containerName="revoke-keys" Oct 30 18:29:36.021779 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.021726 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="482da54b-63ab-4595-9d44-3ca24ed32d03" containerName="revoke-keys" Oct 30 18:29:36.021914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.021881 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="482da54b-63ab-4595-9d44-3ca24ed32d03" containerName="revoke-keys" Oct 30 18:29:36.021959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.021917 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc39faea-f977-4ba2-be13-cef341667507" containerName="istio-proxy" Oct 30 18:29:36.027927 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.027903 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f" Oct 30 18:29:36.040332 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.040301 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f"] Oct 30 18:29:36.043559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.043539 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:29:36.043939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.043904 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:29:36.210099 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.210060 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe28888-b8fc-419c-a756-d417a58ebd14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f\" (UID: \"fbe28888-b8fc-419c-a756-d417a58ebd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f" Oct 30 18:29:36.210281 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.210252 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72c9b\" (UniqueName: \"kubernetes.io/projected/fbe28888-b8fc-419c-a756-d417a58ebd14-kube-api-access-72c9b\") pod \"maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f\" (UID: \"fbe28888-b8fc-419c-a756-d417a58ebd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f" Oct 30 18:29:36.210386 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.210370 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe28888-b8fc-419c-a756-d417a58ebd14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f\" (UID: \"fbe28888-b8fc-419c-a756-d417a58ebd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f" Oct 30 18:29:36.311747 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.311664 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe28888-b8fc-419c-a756-d417a58ebd14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f\" (UID: \"fbe28888-b8fc-419c-a756-d417a58ebd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f" Oct 30 18:29:36.311924 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.311761 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe28888-b8fc-419c-a756-d417a58ebd14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f\" (UID: \"fbe28888-b8fc-419c-a756-d417a58ebd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f" Oct 30 18:29:36.311924 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.311902 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72c9b\" (UniqueName: \"kubernetes.io/projected/fbe28888-b8fc-419c-a756-d417a58ebd14-kube-api-access-72c9b\") pod \"maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f\" (UID: \"fbe28888-b8fc-419c-a756-d417a58ebd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f" Oct 30 18:29:36.312450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.312422 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe28888-b8fc-419c-a756-d417a58ebd14-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f\" (UID: \"fbe28888-b8fc-419c-a756-d417a58ebd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f" Oct 30 18:29:36.314240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.314221 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe28888-b8fc-419c-a756-d417a58ebd14-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f\" (UID: \"fbe28888-b8fc-419c-a756-d417a58ebd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f" Oct 30 18:29:36.331303 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.331277 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-72c9b\" (UniqueName: \"kubernetes.io/projected/fbe28888-b8fc-419c-a756-d417a58ebd14-kube-api-access-72c9b\") pod \"maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f\" (UID: \"fbe28888-b8fc-419c-a756-d417a58ebd14\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f" Oct 30 18:29:36.352261 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.352234 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f" Oct 30 18:29:36.674109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:36.674080 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f"] Oct 30 18:29:36.674325 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:29:36.674216 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe28888_b8fc_419c_a756_d417a58ebd14.slice/crio-7d481f7b35b8bfae3d128169e6351f26129ac1ffc780e056a312479130a9904b WatchSource:0}: Error finding container 7d481f7b35b8bfae3d128169e6351f26129ac1ffc780e056a312479130a9904b: Status 404 returned error can't find the container with id 7d481f7b35b8bfae3d128169e6351f26129ac1ffc780e056a312479130a9904b Oct 30 18:29:37.386650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:37.386617 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f" event={"ID":"fbe28888-b8fc-419c-a756-d417a58ebd14","Type":"ContainerStarted","Data":"be247825d58d37067a528577f5cf3bf1be1d100c4221c3b2b17d8506607795d4"} Oct 30 18:29:37.387139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:37.386666 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f" event={"ID":"fbe28888-b8fc-419c-a756-d417a58ebd14","Type":"ContainerStarted","Data":"7d481f7b35b8bfae3d128169e6351f26129ac1ffc780e056a312479130a9904b"} Oct 30 18:29:37.388464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:37.388439 5232 generic.go:356] "Generic (PLEG): container finished" podID="de317989-41ca-479f-bcc1-9c7b5a166e70" containerID="23575259bdca43d9ceb0b2822da983b9abd29ebdf649da0ce863a0f7385a4b6a" exitCode=0 Oct 30 18:29:37.388619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:37.388503 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" event={"ID":"de317989-41ca-479f-bcc1-9c7b5a166e70","Type":"ContainerDied","Data":"23575259bdca43d9ceb0b2822da983b9abd29ebdf649da0ce863a0f7385a4b6a"} Oct 30 18:29:37.388881 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:37.388867 5232 scope.go:122] "RemoveContainer" containerID="23575259bdca43d9ceb0b2822da983b9abd29ebdf649da0ce863a0f7385a4b6a" Oct 30 18:29:37.390259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:37.390230 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:29:37.390344 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:37.390300 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:29:37.390344 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:37.390336 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:29:37.428160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:37.428097 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f" podStartSLOduration=2.428077817 podStartE2EDuration="2.428077817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:37.425241193 +0000 UTC m=+1676.741661516" watchObservedRunningTime="2026-10-30 18:29:37.428077817 +0000 UTC m=+1676.744498139" Oct 30 18:29:38.397729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:38.397694 5232 generic.go:356] "Generic (PLEG): container finished" podID="fbe28888-b8fc-419c-a756-d417a58ebd14" containerID="be247825d58d37067a528577f5cf3bf1be1d100c4221c3b2b17d8506607795d4" exitCode=0 Oct 30 18:29:38.398141 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:38.397781 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f" event={"ID":"fbe28888-b8fc-419c-a756-d417a58ebd14","Type":"ContainerDied","Data":"be247825d58d37067a528577f5cf3bf1be1d100c4221c3b2b17d8506607795d4"} Oct 30 18:29:38.400053 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:38.400026 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" event={"ID":"de317989-41ca-479f-bcc1-9c7b5a166e70","Type":"ContainerStarted","Data":"381dd6b472d3c1be4ad5a54e72486222ba4ab85cf75b4ba876952f77f1e138b8"} Oct 30 18:29:39.138952 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:39.138914 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:29:39.140166 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:39.140140 5232 patch_prober.go:29] interesting pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 18:29:39.140308 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:39.140189 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" podUID="de317989-41ca-479f-bcc1-9c7b5a166e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 18:29:39.650865 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:39.650841 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f" Oct 30 18:29:39.662870 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:39.662842 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fbe28888-b8fc-419c-a756-d417a58ebd14-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe28888-b8fc-419c-a756-d417a58ebd14-maas-api-xks-ca\") pod \"fbe28888-b8fc-419c-a756-d417a58ebd14\" (UID: \"fbe28888-b8fc-419c-a756-d417a58ebd14\") " Oct 30 18:29:39.663011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:39.662931 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fbe28888-b8fc-419c-a756-d417a58ebd14-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe28888-b8fc-419c-a756-d417a58ebd14-maas-api-openshift-ca\") pod \"fbe28888-b8fc-419c-a756-d417a58ebd14\" (UID: \"fbe28888-b8fc-419c-a756-d417a58ebd14\") " Oct 30 18:29:39.663109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:39.663095 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbe28888-b8fc-419c-a756-d417a58ebd14-kube-api-access-72c9b\" (UniqueName: \"kubernetes.io/projected/fbe28888-b8fc-419c-a756-d417a58ebd14-kube-api-access-72c9b\") pod \"fbe28888-b8fc-419c-a756-d417a58ebd14\" (UID: \"fbe28888-b8fc-419c-a756-d417a58ebd14\") " Oct 30 18:29:39.663261 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:39.663223 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fbe28888-b8fc-419c-a756-d417a58ebd14-maas-api-openshift-ca" pod "fbe28888-b8fc-419c-a756-d417a58ebd14" (UID: "fbe28888-b8fc-419c-a756-d417a58ebd14"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:39.663480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:39.663464 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fbe28888-b8fc-419c-a756-d417a58ebd14-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:39.665244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:39.665222 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbe28888-b8fc-419c-a756-d417a58ebd14-kube-api-access-72c9b" pod "fbe28888-b8fc-419c-a756-d417a58ebd14" (UID: "fbe28888-b8fc-419c-a756-d417a58ebd14"). InnerVolumeSpecName "kube-api-access-72c9b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:39.665418 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:39.665395 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fbe28888-b8fc-419c-a756-d417a58ebd14-maas-api-xks-ca" pod "fbe28888-b8fc-419c-a756-d417a58ebd14" (UID: "fbe28888-b8fc-419c-a756-d417a58ebd14"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:39.775907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:39.764437 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fbe28888-b8fc-419c-a756-d417a58ebd14-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:39.775907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:39.764473 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-72c9b\" (UniqueName: \"kubernetes.io/projected/fbe28888-b8fc-419c-a756-d417a58ebd14-kube-api-access-72c9b\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:39.813702 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:39.813588 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" podUID="fc97fed8-9a76-4696-ab34-5a4a1d401ac3" containerName="payload-processing" containerID="cri-o://4a9a7043a7f3301f1aea78e41a0f5f2d48f5deb104e8c7651932d6ed2e319fff" gracePeriod=25 Oct 30 18:29:39.847381 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:39.847121 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" podUID="9433bec1-83df-429a-98a3-4b573d4fb5f6" containerName="payload-pre-processing" containerID="cri-o://c03ea0f6812a8a607cf8f7f10635cfc832de40a11e0d0d31c6d2fd51817355fd" gracePeriod=25 Oct 30 18:29:40.139188 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:40.139091 5232 patch_prober.go:29] interesting pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 18:29:40.139188 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:40.139162 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" podUID="de317989-41ca-479f-bcc1-9c7b5a166e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 18:29:40.450062 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:40.449969 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f" Oct 30 18:29:40.450062 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:40.449985 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f" event={"ID":"fbe28888-b8fc-419c-a756-d417a58ebd14","Type":"ContainerDied","Data":"7d481f7b35b8bfae3d128169e6351f26129ac1ffc780e056a312479130a9904b"} Oct 30 18:29:40.450062 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:40.450050 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d481f7b35b8bfae3d128169e6351f26129ac1ffc780e056a312479130a9904b" Oct 30 18:29:40.508618 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:29:40.508589 5232 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbe28888_b8fc_419c_a756_d417a58ebd14.slice\": RecentStats: unable to find data in memory cache]" Oct 30 18:29:40.758688 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:40.758655 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz"] Oct 30 18:29:40.770874 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:40.770842 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz"] Oct 30 18:29:40.778628 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:40.778587 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k"] Oct 30 18:29:41.139216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:41.139129 5232 patch_prober.go:29] interesting pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 18:29:41.139216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:41.139197 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" podUID="de317989-41ca-479f-bcc1-9c7b5a166e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 18:29:41.940542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:41.940103 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj"] Oct 30 18:29:41.941065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:41.941034 5232 cpu_manager.go:414] "containerMap: removing container" podUID="fbe28888-b8fc-419c-a756-d417a58ebd14" containerName="revoke-keys" Oct 30 18:29:41.941065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:41.941063 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbe28888-b8fc-419c-a756-d417a58ebd14" containerName="revoke-keys" Oct 30 18:29:41.941309 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:41.941246 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbe28888-b8fc-419c-a756-d417a58ebd14" containerName="revoke-keys" Oct 30 18:29:41.944543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:41.944487 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj" Oct 30 18:29:41.945018 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:41.944997 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs"] Oct 30 18:29:41.948205 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:41.948184 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs" Oct 30 18:29:41.948797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:41.948769 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj"] Oct 30 18:29:41.959327 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:41.959298 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs"] Oct 30 18:29:41.959768 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:41.959740 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:29:41.959899 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:41.959800 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:29:42.103841 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.103796 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-531d7f2f-675vs\" (UID: \"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs" Oct 30 18:29:42.104031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.103966 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/476a0b93-3e62-4ae7-8287-4f620e7e016b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d67609-vl9cj\" (UID: \"476a0b93-3e62-4ae7-8287-4f620e7e016b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj" Oct 30 18:29:42.104098 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.104049 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/476a0b93-3e62-4ae7-8287-4f620e7e016b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d67609-vl9cj\" (UID: \"476a0b93-3e62-4ae7-8287-4f620e7e016b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj" Oct 30 18:29:42.104098 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.104085 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8bxkr\" (UniqueName: \"kubernetes.io/projected/476a0b93-3e62-4ae7-8287-4f620e7e016b-kube-api-access-8bxkr\") pod \"maas-api-revoke-sub-01d67609-vl9cj\" (UID: \"476a0b93-3e62-4ae7-8287-4f620e7e016b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj" Oct 30 18:29:42.104210 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.104164 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcbhk\" (UniqueName: \"kubernetes.io/projected/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-kube-api-access-pcbhk\") pod \"maas-api-revoke-sub-531d7f2f-675vs\" (UID: \"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs" Oct 30 18:29:42.104264 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.104245 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-531d7f2f-675vs\" (UID: \"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs" Oct 30 18:29:42.139092 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.139051 5232 patch_prober.go:29] interesting pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 18:29:42.139305 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.139133 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" podUID="de317989-41ca-479f-bcc1-9c7b5a166e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 18:29:42.205960 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.205794 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-531d7f2f-675vs\" (UID: \"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs" Oct 30 18:29:42.205960 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.205932 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/476a0b93-3e62-4ae7-8287-4f620e7e016b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d67609-vl9cj\" (UID: \"476a0b93-3e62-4ae7-8287-4f620e7e016b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj" Oct 30 18:29:42.206207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.206015 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/476a0b93-3e62-4ae7-8287-4f620e7e016b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d67609-vl9cj\" (UID: \"476a0b93-3e62-4ae7-8287-4f620e7e016b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj" Oct 30 18:29:42.206207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.206054 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8bxkr\" (UniqueName: \"kubernetes.io/projected/476a0b93-3e62-4ae7-8287-4f620e7e016b-kube-api-access-8bxkr\") pod \"maas-api-revoke-sub-01d67609-vl9cj\" (UID: \"476a0b93-3e62-4ae7-8287-4f620e7e016b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj" Oct 30 18:29:42.206207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.206102 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcbhk\" (UniqueName: \"kubernetes.io/projected/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-kube-api-access-pcbhk\") pod \"maas-api-revoke-sub-531d7f2f-675vs\" (UID: \"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs" Oct 30 18:29:42.206207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.206150 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-531d7f2f-675vs\" (UID: \"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs" Oct 30 18:29:42.206856 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.206823 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-531d7f2f-675vs\" (UID: \"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs" Oct 30 18:29:42.206963 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.206918 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/476a0b93-3e62-4ae7-8287-4f620e7e016b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-01d67609-vl9cj\" (UID: \"476a0b93-3e62-4ae7-8287-4f620e7e016b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj" Oct 30 18:29:42.208604 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.208583 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/476a0b93-3e62-4ae7-8287-4f620e7e016b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-01d67609-vl9cj\" (UID: \"476a0b93-3e62-4ae7-8287-4f620e7e016b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj" Oct 30 18:29:42.208693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.208602 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-531d7f2f-675vs\" (UID: \"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs" Oct 30 18:29:42.238859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.238818 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcbhk\" (UniqueName: \"kubernetes.io/projected/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-kube-api-access-pcbhk\") pod \"maas-api-revoke-sub-531d7f2f-675vs\" (UID: \"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs" Oct 30 18:29:42.239602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.239574 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8bxkr\" (UniqueName: \"kubernetes.io/projected/476a0b93-3e62-4ae7-8287-4f620e7e016b-kube-api-access-8bxkr\") pod \"maas-api-revoke-sub-01d67609-vl9cj\" (UID: \"476a0b93-3e62-4ae7-8287-4f620e7e016b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj" Oct 30 18:29:42.266556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.266496 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj" Oct 30 18:29:42.278733 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.278705 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs" Oct 30 18:29:42.601779 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.601748 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj"] Oct 30 18:29:42.602783 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:29:42.602752 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod476a0b93_3e62_4ae7_8287_4f620e7e016b.slice/crio-097fa9738bbec58b5897eb5afbd5bbf699652ec4f203d3ee00e2cbc7b38349d6 WatchSource:0}: Error finding container 097fa9738bbec58b5897eb5afbd5bbf699652ec4f203d3ee00e2cbc7b38349d6: Status 404 returned error can't find the container with id 097fa9738bbec58b5897eb5afbd5bbf699652ec4f203d3ee00e2cbc7b38349d6 Oct 30 18:29:42.641286 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:29:42.639750 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e6476fd_f31d_4f29_a5c7_4e51df8ff9f0.slice/crio-f0f694b52f7f0a81ff1a7e3c053f1f7d74fe87adc7c2697d6f98d179c8ae95d9 WatchSource:0}: Error finding container f0f694b52f7f0a81ff1a7e3c053f1f7d74fe87adc7c2697d6f98d179c8ae95d9: Status 404 returned error can't find the container with id f0f694b52f7f0a81ff1a7e3c053f1f7d74fe87adc7c2697d6f98d179c8ae95d9 Oct 30 18:29:42.642420 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.642396 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs"] Oct 30 18:29:42.936581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.936463 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" podUID="9433bec1-83df-429a-98a3-4b573d4fb5f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:29:42.936846 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:42.936686 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" podUID="fc97fed8-9a76-4696-ab34-5a4a1d401ac3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:29:43.138404 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:43.138378 5232 patch_prober.go:29] interesting pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" start-of-body= Oct 30 18:29:43.138876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:43.138425 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" podUID="de317989-41ca-479f-bcc1-9c7b5a166e70" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.210:15021/healthz/ready\": dial tcp 10.133.0.210:15021: connect: connection refused" Oct 30 18:29:43.256191 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:43.256152 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:29:43.503951 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:43.503915 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs" event={"ID":"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0","Type":"ContainerStarted","Data":"10e77edb2df73cdb870bcd223e9b0f683be5da908a21c1e7564b6185c076322c"} Oct 30 18:29:43.504156 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:43.503975 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs" event={"ID":"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0","Type":"ContainerStarted","Data":"f0f694b52f7f0a81ff1a7e3c053f1f7d74fe87adc7c2697d6f98d179c8ae95d9"} Oct 30 18:29:43.505873 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:43.505848 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj" event={"ID":"476a0b93-3e62-4ae7-8287-4f620e7e016b","Type":"ContainerStarted","Data":"a4108ce01f8ce085eb19b6999f585c836690d14f74e7f044cb0e29cfa5bc1106"} Oct 30 18:29:43.505985 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:43.505892 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj" event={"ID":"476a0b93-3e62-4ae7-8287-4f620e7e016b","Type":"ContainerStarted","Data":"097fa9738bbec58b5897eb5afbd5bbf699652ec4f203d3ee00e2cbc7b38349d6"} Oct 30 18:29:43.547117 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:43.547005 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs" podStartSLOduration=2.546984843 podStartE2EDuration="2.546984843s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:43.544646421 +0000 UTC m=+1682.861066744" watchObservedRunningTime="2026-10-30 18:29:43.546984843 +0000 UTC m=+1682.863405165" Oct 30 18:29:43.588321 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:43.588256 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj" podStartSLOduration=2.588237302 podStartE2EDuration="2.588237302s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:29:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:29:43.583493071 +0000 UTC m=+1682.899913396" watchObservedRunningTime="2026-10-30 18:29:43.588237302 +0000 UTC m=+1682.904657624" Oct 30 18:29:44.158532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.158429 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:29:44.162021 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.161986 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:29:44.172710 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.172665 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" containerID="cri-o://56872a09f78362d8951e8a644657fdd967c7a63eb664b4d1715501850a3b6ff6" gracePeriod=45 Oct 30 18:29:44.200913 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.200847 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" containerID="cri-o://adb635b41935ea40e15bca8b12d68c6dd85e96f4fdb06f42b11a84755d464648" gracePeriod=45 Oct 30 18:29:44.338776 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.338735 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" probeResult="failure" output="Get \"https://10.133.0.103:8000/ready\": dial tcp 10.133.0.103:8000: connect: connection refused" Oct 30 18:29:44.476287 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.476257 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:29:44.523198 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.523161 5232 generic.go:356] "Generic (PLEG): container finished" podID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerID="56872a09f78362d8951e8a644657fdd967c7a63eb664b4d1715501850a3b6ff6" exitCode=0 Oct 30 18:29:44.523396 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.523377 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" event={"ID":"d74ee1cf-7d30-49b6-880d-a7368794823d","Type":"ContainerDied","Data":"56872a09f78362d8951e8a644657fdd967c7a63eb664b4d1715501850a3b6ff6"} Oct 30 18:29:44.523473 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.523396 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" Oct 30 18:29:44.523473 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.523417 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd" event={"ID":"d74ee1cf-7d30-49b6-880d-a7368794823d","Type":"ContainerDied","Data":"b5555675f49deac9cb422e983d9792ce541e0d82c5a46f787f7651ff0d1c6f7a"} Oct 30 18:29:44.523473 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.523443 5232 scope.go:122] "RemoveContainer" containerID="56872a09f78362d8951e8a644657fdd967c7a63eb664b4d1715501850a3b6ff6" Oct 30 18:29:44.533563 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.533488 5232 generic.go:356] "Generic (PLEG): container finished" podID="5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0" containerID="10e77edb2df73cdb870bcd223e9b0f683be5da908a21c1e7564b6185c076322c" exitCode=0 Oct 30 18:29:44.533744 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.533720 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs" event={"ID":"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0","Type":"ContainerDied","Data":"10e77edb2df73cdb870bcd223e9b0f683be5da908a21c1e7564b6185c076322c"} Oct 30 18:29:44.536477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.536452 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:29:44.540952 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.540925 5232 generic.go:356] "Generic (PLEG): container finished" podID="476a0b93-3e62-4ae7-8287-4f620e7e016b" containerID="a4108ce01f8ce085eb19b6999f585c836690d14f74e7f044cb0e29cfa5bc1106" exitCode=0 Oct 30 18:29:44.541078 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.541036 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj" event={"ID":"476a0b93-3e62-4ae7-8287-4f620e7e016b","Type":"ContainerDied","Data":"a4108ce01f8ce085eb19b6999f585c836690d14f74e7f044cb0e29cfa5bc1106"} Oct 30 18:29:44.541838 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.541822 5232 scope.go:122] "RemoveContainer" containerID="56872a09f78362d8951e8a644657fdd967c7a63eb664b4d1715501850a3b6ff6" Oct 30 18:29:44.542123 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:29:44.542092 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56872a09f78362d8951e8a644657fdd967c7a63eb664b4d1715501850a3b6ff6\": container with ID starting with 56872a09f78362d8951e8a644657fdd967c7a63eb664b4d1715501850a3b6ff6 not found: ID does not exist" containerID="56872a09f78362d8951e8a644657fdd967c7a63eb664b4d1715501850a3b6ff6" Oct 30 18:29:44.542250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.542133 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56872a09f78362d8951e8a644657fdd967c7a63eb664b4d1715501850a3b6ff6"} err="failed to get container status \"56872a09f78362d8951e8a644657fdd967c7a63eb664b4d1715501850a3b6ff6\": rpc error: code = NotFound desc = could not find container \"56872a09f78362d8951e8a644657fdd967c7a63eb664b4d1715501850a3b6ff6\": container with ID starting with 56872a09f78362d8951e8a644657fdd967c7a63eb664b4d1715501850a3b6ff6 not found: ID does not exist" Oct 30 18:29:44.543078 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.543053 5232 generic.go:356] "Generic (PLEG): container finished" podID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerID="adb635b41935ea40e15bca8b12d68c6dd85e96f4fdb06f42b11a84755d464648" exitCode=0 Oct 30 18:29:44.543195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.543093 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" Oct 30 18:29:44.543195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.543142 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" event={"ID":"05a41b45-1068-426b-bbf8-422e1b4980ba","Type":"ContainerDied","Data":"adb635b41935ea40e15bca8b12d68c6dd85e96f4fdb06f42b11a84755d464648"} Oct 30 18:29:44.543195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.543180 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w" event={"ID":"05a41b45-1068-426b-bbf8-422e1b4980ba","Type":"ContainerDied","Data":"9cf5a7a4fab40952a16071b1a99798c276d9aaaf1ccc8c43b750fda9dc6661fe"} Oct 30 18:29:44.543372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.543220 5232 scope.go:122] "RemoveContainer" containerID="adb635b41935ea40e15bca8b12d68c6dd85e96f4fdb06f42b11a84755d464648" Oct 30 18:29:44.557209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.557187 5232 scope.go:122] "RemoveContainer" containerID="adb635b41935ea40e15bca8b12d68c6dd85e96f4fdb06f42b11a84755d464648" Oct 30 18:29:44.557611 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:29:44.557584 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"adb635b41935ea40e15bca8b12d68c6dd85e96f4fdb06f42b11a84755d464648\": container with ID starting with adb635b41935ea40e15bca8b12d68c6dd85e96f4fdb06f42b11a84755d464648 not found: ID does not exist" containerID="adb635b41935ea40e15bca8b12d68c6dd85e96f4fdb06f42b11a84755d464648" Oct 30 18:29:44.557692 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.557623 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"adb635b41935ea40e15bca8b12d68c6dd85e96f4fdb06f42b11a84755d464648"} err="failed to get container status \"adb635b41935ea40e15bca8b12d68c6dd85e96f4fdb06f42b11a84755d464648\": rpc error: code = NotFound desc = could not find container \"adb635b41935ea40e15bca8b12d68c6dd85e96f4fdb06f42b11a84755d464648\": container with ID starting with adb635b41935ea40e15bca8b12d68c6dd85e96f4fdb06f42b11a84755d464648 not found: ID does not exist" Oct 30 18:29:44.570848 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.570821 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-tmp-dir\") pod \"05a41b45-1068-426b-bbf8-422e1b4980ba\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " Oct 30 18:29:44.571003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.570862 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-model-cache\") pod \"d74ee1cf-7d30-49b6-880d-a7368794823d\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " Oct 30 18:29:44.571003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.570889 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-model-cache\") pod \"05a41b45-1068-426b-bbf8-422e1b4980ba\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " Oct 30 18:29:44.571003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.570924 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d74ee1cf-7d30-49b6-880d-a7368794823d-kube-api-access-wcw4z\" (UniqueName: \"kubernetes.io/projected/d74ee1cf-7d30-49b6-880d-a7368794823d-kube-api-access-wcw4z\") pod \"d74ee1cf-7d30-49b6-880d-a7368794823d\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " Oct 30 18:29:44.571003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.570992 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-dshm\") pod \"d74ee1cf-7d30-49b6-880d-a7368794823d\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " Oct 30 18:29:44.571214 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.571024 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05a41b45-1068-426b-bbf8-422e1b4980ba-tls-certs\" (UniqueName: \"kubernetes.io/secret/05a41b45-1068-426b-bbf8-422e1b4980ba-tls-certs\") pod \"05a41b45-1068-426b-bbf8-422e1b4980ba\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " Oct 30 18:29:44.571214 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.571069 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d74ee1cf-7d30-49b6-880d-a7368794823d-tls-certs\" (UniqueName: \"kubernetes.io/secret/d74ee1cf-7d30-49b6-880d-a7368794823d-tls-certs\") pod \"d74ee1cf-7d30-49b6-880d-a7368794823d\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " Oct 30 18:29:44.571214 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.571100 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-tmp-dir" pod "05a41b45-1068-426b-bbf8-422e1b4980ba" (UID: "05a41b45-1068-426b-bbf8-422e1b4980ba"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:44.571214 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.571141 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-home\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-home\") pod \"d74ee1cf-7d30-49b6-880d-a7368794823d\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " Oct 30 18:29:44.571411 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.571219 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-model-cache" pod "d74ee1cf-7d30-49b6-880d-a7368794823d" (UID: "d74ee1cf-7d30-49b6-880d-a7368794823d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:44.571411 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.571255 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-model-cache" pod "05a41b45-1068-426b-bbf8-422e1b4980ba" (UID: "05a41b45-1068-426b-bbf8-422e1b4980ba"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:44.571411 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.571282 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-tmp-dir\") pod \"d74ee1cf-7d30-49b6-880d-a7368794823d\" (UID: \"d74ee1cf-7d30-49b6-880d-a7368794823d\") " Oct 30 18:29:44.571411 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.571337 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-home\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-home\") pod \"05a41b45-1068-426b-bbf8-422e1b4980ba\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " Oct 30 18:29:44.571411 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.571389 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-dshm\") pod \"05a41b45-1068-426b-bbf8-422e1b4980ba\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " Oct 30 18:29:44.571718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.571451 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05a41b45-1068-426b-bbf8-422e1b4980ba-kube-api-access-7fltj\" (UniqueName: \"kubernetes.io/projected/05a41b45-1068-426b-bbf8-422e1b4980ba-kube-api-access-7fltj\") pod \"05a41b45-1068-426b-bbf8-422e1b4980ba\" (UID: \"05a41b45-1068-426b-bbf8-422e1b4980ba\") " Oct 30 18:29:44.571718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.571676 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-home" pod "d74ee1cf-7d30-49b6-880d-a7368794823d" (UID: "d74ee1cf-7d30-49b6-880d-a7368794823d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:44.571928 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.571886 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-home" pod "05a41b45-1068-426b-bbf8-422e1b4980ba" (UID: "05a41b45-1068-426b-bbf8-422e1b4980ba"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:44.572198 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.572170 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-tmp-dir" pod "d74ee1cf-7d30-49b6-880d-a7368794823d" (UID: "d74ee1cf-7d30-49b6-880d-a7368794823d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:44.572343 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.572321 5232 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-home\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.572423 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.572350 5232 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-tmp-dir\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.572423 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.572366 5232 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-home\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.572423 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.572379 5232 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-tmp-dir\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.572423 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.572392 5232 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-model-cache\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.572423 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.572405 5232 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-model-cache\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.575139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.575117 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05a41b45-1068-426b-bbf8-422e1b4980ba-kube-api-access-7fltj" pod "05a41b45-1068-426b-bbf8-422e1b4980ba" (UID: "05a41b45-1068-426b-bbf8-422e1b4980ba"). InnerVolumeSpecName "kube-api-access-7fltj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:44.575245 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.575183 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-dshm" pod "05a41b45-1068-426b-bbf8-422e1b4980ba" (UID: "05a41b45-1068-426b-bbf8-422e1b4980ba"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:44.575883 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.575857 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-dshm" pod "d74ee1cf-7d30-49b6-880d-a7368794823d" (UID: "d74ee1cf-7d30-49b6-880d-a7368794823d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:44.575969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.575891 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d74ee1cf-7d30-49b6-880d-a7368794823d-tls-certs" pod "d74ee1cf-7d30-49b6-880d-a7368794823d" (UID: "d74ee1cf-7d30-49b6-880d-a7368794823d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:44.577842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.576524 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05a41b45-1068-426b-bbf8-422e1b4980ba-tls-certs" pod "05a41b45-1068-426b-bbf8-422e1b4980ba" (UID: "05a41b45-1068-426b-bbf8-422e1b4980ba"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:44.579240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.579213 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d74ee1cf-7d30-49b6-880d-a7368794823d-kube-api-access-wcw4z" pod "d74ee1cf-7d30-49b6-880d-a7368794823d" (UID: "d74ee1cf-7d30-49b6-880d-a7368794823d"). InnerVolumeSpecName "kube-api-access-wcw4z". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:44.673976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.673943 5232 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d74ee1cf-7d30-49b6-880d-a7368794823d-dshm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.673976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.673971 5232 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/05a41b45-1068-426b-bbf8-422e1b4980ba-tls-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.673976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.673982 5232 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d74ee1cf-7d30-49b6-880d-a7368794823d-tls-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.674292 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.673990 5232 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/05a41b45-1068-426b-bbf8-422e1b4980ba-dshm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.674292 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.673998 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fltj\" (UniqueName: \"kubernetes.io/projected/05a41b45-1068-426b-bbf8-422e1b4980ba-kube-api-access-7fltj\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.674292 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.674007 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wcw4z\" (UniqueName: \"kubernetes.io/projected/d74ee1cf-7d30-49b6-880d-a7368794823d-kube-api-access-wcw4z\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:44.864013 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.863989 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd"] Oct 30 18:29:44.867144 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.867123 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/distinct-2-w1-fb2fa0-kserve-67c56cd94b-wqrhd"] Oct 30 18:29:44.900115 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.900082 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w"] Oct 30 18:29:44.903316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:44.903299 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-fb2fa0/simulator-w1-fb2fa0-kserve-6b6c7bdd79-kwn9w"] Oct 30 18:29:45.764562 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.764527 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs" Oct 30 18:29:45.789058 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.789030 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-maas-api-openshift-ca\") pod \"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0\" (UID: \"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0\") " Oct 30 18:29:45.789249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.789225 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-maas-api-xks-ca\") pod \"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0\" (UID: \"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0\") " Oct 30 18:29:45.789387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.789368 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-kube-api-access-pcbhk\" (UniqueName: \"kubernetes.io/projected/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-kube-api-access-pcbhk\") pod \"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0\" (UID: \"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0\") " Oct 30 18:29:45.789471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.789379 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-maas-api-openshift-ca" pod "5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0" (UID: "5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:45.789957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.789930 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:45.791709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.791685 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-maas-api-xks-ca" pod "5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0" (UID: "5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:45.791831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.791781 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-kube-api-access-pcbhk" pod "5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0" (UID: "5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0"). InnerVolumeSpecName "kube-api-access-pcbhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:45.817587 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.817555 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj" Oct 30 18:29:45.890838 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.890796 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/476a0b93-3e62-4ae7-8287-4f620e7e016b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/476a0b93-3e62-4ae7-8287-4f620e7e016b-maas-api-xks-ca\") pod \"476a0b93-3e62-4ae7-8287-4f620e7e016b\" (UID: \"476a0b93-3e62-4ae7-8287-4f620e7e016b\") " Oct 30 18:29:45.891051 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.890913 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/476a0b93-3e62-4ae7-8287-4f620e7e016b-kube-api-access-8bxkr\" (UniqueName: \"kubernetes.io/projected/476a0b93-3e62-4ae7-8287-4f620e7e016b-kube-api-access-8bxkr\") pod \"476a0b93-3e62-4ae7-8287-4f620e7e016b\" (UID: \"476a0b93-3e62-4ae7-8287-4f620e7e016b\") " Oct 30 18:29:45.891051 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.891044 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/476a0b93-3e62-4ae7-8287-4f620e7e016b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/476a0b93-3e62-4ae7-8287-4f620e7e016b-maas-api-openshift-ca\") pod \"476a0b93-3e62-4ae7-8287-4f620e7e016b\" (UID: \"476a0b93-3e62-4ae7-8287-4f620e7e016b\") " Oct 30 18:29:45.891425 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.891396 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/476a0b93-3e62-4ae7-8287-4f620e7e016b-maas-api-openshift-ca" pod "476a0b93-3e62-4ae7-8287-4f620e7e016b" (UID: "476a0b93-3e62-4ae7-8287-4f620e7e016b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:29:45.891915 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.891843 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/476a0b93-3e62-4ae7-8287-4f620e7e016b-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:45.891915 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.891870 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:45.891915 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.891885 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pcbhk\" (UniqueName: \"kubernetes.io/projected/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0-kube-api-access-pcbhk\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:45.893411 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.893322 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/476a0b93-3e62-4ae7-8287-4f620e7e016b-kube-api-access-8bxkr" pod "476a0b93-3e62-4ae7-8287-4f620e7e016b" (UID: "476a0b93-3e62-4ae7-8287-4f620e7e016b"). InnerVolumeSpecName "kube-api-access-8bxkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:45.893491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.893423 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/476a0b93-3e62-4ae7-8287-4f620e7e016b-maas-api-xks-ca" pod "476a0b93-3e62-4ae7-8287-4f620e7e016b" (UID: "476a0b93-3e62-4ae7-8287-4f620e7e016b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:45.993310 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.993276 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8bxkr\" (UniqueName: \"kubernetes.io/projected/476a0b93-3e62-4ae7-8287-4f620e7e016b-kube-api-access-8bxkr\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:45.993310 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:45.993306 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/476a0b93-3e62-4ae7-8287-4f620e7e016b-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:46.519573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:46.519484 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6"] Oct 30 18:29:46.562145 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:46.562119 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs" Oct 30 18:29:46.562363 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:46.562160 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs" event={"ID":"5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0","Type":"ContainerDied","Data":"f0f694b52f7f0a81ff1a7e3c053f1f7d74fe87adc7c2697d6f98d179c8ae95d9"} Oct 30 18:29:46.562363 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:46.562212 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0f694b52f7f0a81ff1a7e3c053f1f7d74fe87adc7c2697d6f98d179c8ae95d9" Oct 30 18:29:46.563600 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:46.563579 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj" Oct 30 18:29:46.563740 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:46.563612 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj" event={"ID":"476a0b93-3e62-4ae7-8287-4f620e7e016b","Type":"ContainerDied","Data":"097fa9738bbec58b5897eb5afbd5bbf699652ec4f203d3ee00e2cbc7b38349d6"} Oct 30 18:29:46.563740 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:46.563668 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="097fa9738bbec58b5897eb5afbd5bbf699652ec4f203d3ee00e2cbc7b38349d6" Oct 30 18:29:46.847441 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:46.847355 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" path="/var/lib/kubelet/pods/05a41b45-1068-426b-bbf8-422e1b4980ba/volumes" Oct 30 18:29:46.848074 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:46.848051 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" path="/var/lib/kubelet/pods/d74ee1cf-7d30-49b6-880d-a7368794823d/volumes" Oct 30 18:29:51.520249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:51.520192 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" podUID="47a6c676-1f34-4d08-81e9-80051001aa80" containerName="payload-processing" containerID="cri-o://3aebff89f6fffffa889ccfff11a77b394bedcb71289ba904029454a339bf8385" gracePeriod=25 Oct 30 18:29:52.930739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:52.930701 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" podUID="9433bec1-83df-429a-98a3-4b573d4fb5f6" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:29:52.931130 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:52.930703 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" podUID="fc97fed8-9a76-4696-ab34-5a4a1d401ac3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:29:54.635545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:54.635494 5232 scope.go:122] "RemoveContainer" containerID="08dfaf70bcfe59935079d5ba74e77a5e41ed0ecb2a97d26de789f84f1c4469df" Oct 30 18:29:55.781503 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:55.781459 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" containerID="cri-o://8aeabf78f92ea63322770ddcba8644e2bc0939d47e9ce78e94c579eae24513ec" gracePeriod=45 Oct 30 18:29:55.794962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:55.794922 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" containerID="cri-o://b0332b10b4ab6eb47cd6dccae366dc9a91d7bb2c384857f58eae8214ca10b7a0" gracePeriod=45 Oct 30 18:29:55.799698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:55.799669 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" containerID="cri-o://b81d1ff926caddabf97a8403c2ef6a8a4fc4681483a8948773e9cf2e63574061" gracePeriod=45 Oct 30 18:29:55.975900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:55.975743 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" podUID="47a6c676-1f34-4d08-81e9-80051001aa80" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:29:56.140271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.140244 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:29:56.148089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.148061 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:29:56.152792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.152763 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:29:56.267435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.267403 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-dshm\") pod \"e70bbbb7-6e09-46cf-9119-868fc52bb412\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " Oct 30 18:29:56.267656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.267448 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-model-cache\") pod \"e70bbbb7-6e09-46cf-9119-868fc52bb412\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " Oct 30 18:29:56.267656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.267537 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e70bbbb7-6e09-46cf-9119-868fc52bb412-kube-api-access-grnvd\" (UniqueName: \"kubernetes.io/projected/e70bbbb7-6e09-46cf-9119-868fc52bb412-kube-api-access-grnvd\") pod \"e70bbbb7-6e09-46cf-9119-868fc52bb412\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " Oct 30 18:29:56.267656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.267583 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-home\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-home\") pod \"e70bbbb7-6e09-46cf-9119-868fc52bb412\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " Oct 30 18:29:56.267656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.267613 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1a0e1051-f784-4141-a8fa-c5fb432f4abd-tls-certs\" (UniqueName: \"kubernetes.io/secret/1a0e1051-f784-4141-a8fa-c5fb432f4abd-tls-certs\") pod \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " Oct 30 18:29:56.267906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.267675 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-dshm\") pod \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " Oct 30 18:29:56.267906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.267730 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ade2fbb-e0fe-4957-a7ff-a74990256474-tls-certs\" (UniqueName: \"kubernetes.io/secret/1ade2fbb-e0fe-4957-a7ff-a74990256474-tls-certs\") pod \"1ade2fbb-e0fe-4957-a7ff-a74990256474\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " Oct 30 18:29:56.267906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.267773 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-tmp-dir\") pod \"e70bbbb7-6e09-46cf-9119-868fc52bb412\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " Oct 30 18:29:56.267906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.267784 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-model-cache" pod "e70bbbb7-6e09-46cf-9119-868fc52bb412" (UID: "e70bbbb7-6e09-46cf-9119-868fc52bb412"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:56.267906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.267828 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-dshm\") pod \"1ade2fbb-e0fe-4957-a7ff-a74990256474\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " Oct 30 18:29:56.267906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.267894 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-home\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-home\") pod \"1ade2fbb-e0fe-4957-a7ff-a74990256474\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " Oct 30 18:29:56.268187 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.267928 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-model-cache\") pod \"1ade2fbb-e0fe-4957-a7ff-a74990256474\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " Oct 30 18:29:56.268187 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.267940 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-home" pod "e70bbbb7-6e09-46cf-9119-868fc52bb412" (UID: "e70bbbb7-6e09-46cf-9119-868fc52bb412"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:56.268187 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.268004 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a0e1051-f784-4141-a8fa-c5fb432f4abd-kube-api-access-tjr57\" (UniqueName: \"kubernetes.io/projected/1a0e1051-f784-4141-a8fa-c5fb432f4abd-kube-api-access-tjr57\") pod \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " Oct 30 18:29:56.268187 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.268082 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-tmp-dir\") pod \"1ade2fbb-e0fe-4957-a7ff-a74990256474\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " Oct 30 18:29:56.268187 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.268126 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-model-cache\") pod \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " Oct 30 18:29:56.268187 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.268161 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-tmp-dir\") pod \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " Oct 30 18:29:56.268428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.268206 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ade2fbb-e0fe-4957-a7ff-a74990256474-kube-api-access-xhmsq\" (UniqueName: \"kubernetes.io/projected/1ade2fbb-e0fe-4957-a7ff-a74990256474-kube-api-access-xhmsq\") pod \"1ade2fbb-e0fe-4957-a7ff-a74990256474\" (UID: \"1ade2fbb-e0fe-4957-a7ff-a74990256474\") " Oct 30 18:29:56.268428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.268240 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-home\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-home\") pod \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\" (UID: \"1a0e1051-f784-4141-a8fa-c5fb432f4abd\") " Oct 30 18:29:56.268428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.268271 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e70bbbb7-6e09-46cf-9119-868fc52bb412-tls-certs\" (UniqueName: \"kubernetes.io/secret/e70bbbb7-6e09-46cf-9119-868fc52bb412-tls-certs\") pod \"e70bbbb7-6e09-46cf-9119-868fc52bb412\" (UID: \"e70bbbb7-6e09-46cf-9119-868fc52bb412\") " Oct 30 18:29:56.268923 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.268899 5232 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-home\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.268923 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.268921 5232 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-model-cache\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.270978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.270739 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-dshm" pod "1a0e1051-f784-4141-a8fa-c5fb432f4abd" (UID: "1a0e1051-f784-4141-a8fa-c5fb432f4abd"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:56.271324 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.271061 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-tmp-dir" pod "1a0e1051-f784-4141-a8fa-c5fb432f4abd" (UID: "1a0e1051-f784-4141-a8fa-c5fb432f4abd"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:56.271324 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.271276 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-tmp-dir" pod "1ade2fbb-e0fe-4957-a7ff-a74990256474" (UID: "1ade2fbb-e0fe-4957-a7ff-a74990256474"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:56.271838 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.271781 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-model-cache" pod "1a0e1051-f784-4141-a8fa-c5fb432f4abd" (UID: "1a0e1051-f784-4141-a8fa-c5fb432f4abd"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:56.273642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.273616 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e70bbbb7-6e09-46cf-9119-868fc52bb412-kube-api-access-grnvd" pod "e70bbbb7-6e09-46cf-9119-868fc52bb412" (UID: "e70bbbb7-6e09-46cf-9119-868fc52bb412"). InnerVolumeSpecName "kube-api-access-grnvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:56.273833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.273811 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e70bbbb7-6e09-46cf-9119-868fc52bb412-tls-certs" pod "e70bbbb7-6e09-46cf-9119-868fc52bb412" (UID: "e70bbbb7-6e09-46cf-9119-868fc52bb412"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:56.274376 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.274345 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ade2fbb-e0fe-4957-a7ff-a74990256474-kube-api-access-xhmsq" pod "1ade2fbb-e0fe-4957-a7ff-a74990256474" (UID: "1ade2fbb-e0fe-4957-a7ff-a74990256474"). InnerVolumeSpecName "kube-api-access-xhmsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:56.274599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.274575 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-model-cache" pod "1ade2fbb-e0fe-4957-a7ff-a74990256474" (UID: "1ade2fbb-e0fe-4957-a7ff-a74990256474"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:56.275129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.275104 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-home" pod "1a0e1051-f784-4141-a8fa-c5fb432f4abd" (UID: "1a0e1051-f784-4141-a8fa-c5fb432f4abd"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:56.275382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.275359 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-tmp-dir" pod "e70bbbb7-6e09-46cf-9119-868fc52bb412" (UID: "e70bbbb7-6e09-46cf-9119-868fc52bb412"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:56.275551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.275525 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-home" pod "1ade2fbb-e0fe-4957-a7ff-a74990256474" (UID: "1ade2fbb-e0fe-4957-a7ff-a74990256474"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:56.276748 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.276361 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-dshm" pod "1ade2fbb-e0fe-4957-a7ff-a74990256474" (UID: "1ade2fbb-e0fe-4957-a7ff-a74990256474"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:56.276839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.276748 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a0e1051-f784-4141-a8fa-c5fb432f4abd-tls-certs" pod "1a0e1051-f784-4141-a8fa-c5fb432f4abd" (UID: "1a0e1051-f784-4141-a8fa-c5fb432f4abd"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:56.276839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.276773 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-dshm" pod "e70bbbb7-6e09-46cf-9119-868fc52bb412" (UID: "e70bbbb7-6e09-46cf-9119-868fc52bb412"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:29:56.277228 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.277204 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ade2fbb-e0fe-4957-a7ff-a74990256474-tls-certs" pod "1ade2fbb-e0fe-4957-a7ff-a74990256474" (UID: "1ade2fbb-e0fe-4957-a7ff-a74990256474"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:29:56.278672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.278642 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a0e1051-f784-4141-a8fa-c5fb432f4abd-kube-api-access-tjr57" pod "1a0e1051-f784-4141-a8fa-c5fb432f4abd" (UID: "1a0e1051-f784-4141-a8fa-c5fb432f4abd"). InnerVolumeSpecName "kube-api-access-tjr57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:29:56.370164 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.370071 5232 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-tmp-dir\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.370164 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.370105 5232 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-model-cache\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.370164 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.370118 5232 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-tmp-dir\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.370164 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.370130 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xhmsq\" (UniqueName: \"kubernetes.io/projected/1ade2fbb-e0fe-4957-a7ff-a74990256474-kube-api-access-xhmsq\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.370164 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.370141 5232 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-home\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.370164 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.370152 5232 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e70bbbb7-6e09-46cf-9119-868fc52bb412-tls-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.370164 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.370164 5232 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-dshm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.370532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.370176 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grnvd\" (UniqueName: \"kubernetes.io/projected/e70bbbb7-6e09-46cf-9119-868fc52bb412-kube-api-access-grnvd\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.370532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.370187 5232 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1a0e1051-f784-4141-a8fa-c5fb432f4abd-tls-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.370532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.370199 5232 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1a0e1051-f784-4141-a8fa-c5fb432f4abd-dshm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.370532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.370209 5232 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1ade2fbb-e0fe-4957-a7ff-a74990256474-tls-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.370532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.370220 5232 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e70bbbb7-6e09-46cf-9119-868fc52bb412-tmp-dir\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.370532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.370230 5232 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-dshm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.370532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.370241 5232 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-home\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.370532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.370252 5232 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1ade2fbb-e0fe-4957-a7ff-a74990256474-model-cache\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.370532 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.370263 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tjr57\" (UniqueName: \"kubernetes.io/projected/1a0e1051-f784-4141-a8fa-c5fb432f4abd-kube-api-access-tjr57\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:29:56.686400 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.686306 5232 generic.go:356] "Generic (PLEG): container finished" podID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerID="b81d1ff926caddabf97a8403c2ef6a8a4fc4681483a8948773e9cf2e63574061" exitCode=0 Oct 30 18:29:56.686400 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.686374 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" Oct 30 18:29:56.686713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.686391 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" event={"ID":"1a0e1051-f784-4141-a8fa-c5fb432f4abd","Type":"ContainerDied","Data":"b81d1ff926caddabf97a8403c2ef6a8a4fc4681483a8948773e9cf2e63574061"} Oct 30 18:29:56.686713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.686443 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" event={"ID":"1a0e1051-f784-4141-a8fa-c5fb432f4abd","Type":"ContainerDied","Data":"2a6e79ad5728b076950067441e2a80cbe10368e226d43e379313cb0c90687705"} Oct 30 18:29:56.686713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.686467 5232 scope.go:122] "RemoveContainer" containerID="b81d1ff926caddabf97a8403c2ef6a8a4fc4681483a8948773e9cf2e63574061" Oct 30 18:29:56.688875 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.688846 5232 generic.go:356] "Generic (PLEG): container finished" podID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerID="b0332b10b4ab6eb47cd6dccae366dc9a91d7bb2c384857f58eae8214ca10b7a0" exitCode=0 Oct 30 18:29:56.689010 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.688882 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" event={"ID":"1ade2fbb-e0fe-4957-a7ff-a74990256474","Type":"ContainerDied","Data":"b0332b10b4ab6eb47cd6dccae366dc9a91d7bb2c384857f58eae8214ca10b7a0"} Oct 30 18:29:56.689010 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.688933 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" Oct 30 18:29:56.689939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.688922 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz" event={"ID":"1ade2fbb-e0fe-4957-a7ff-a74990256474","Type":"ContainerDied","Data":"9238a95bdefc3bce3d3352a10b8237c1450926fdb4853ee0d06307d886484c97"} Oct 30 18:29:56.695078 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.695035 5232 generic.go:356] "Generic (PLEG): container finished" podID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerID="8aeabf78f92ea63322770ddcba8644e2bc0939d47e9ce78e94c579eae24513ec" exitCode=0 Oct 30 18:29:56.695199 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.695111 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" Oct 30 18:29:56.695277 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.695255 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" event={"ID":"e70bbbb7-6e09-46cf-9119-868fc52bb412","Type":"ContainerDied","Data":"8aeabf78f92ea63322770ddcba8644e2bc0939d47e9ce78e94c579eae24513ec"} Oct 30 18:29:56.695334 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.695298 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz" event={"ID":"e70bbbb7-6e09-46cf-9119-868fc52bb412","Type":"ContainerDied","Data":"6c51fdf7e6597f58cc63dec3702015724c25aa7623200b3373dd2eb37329c6fd"} Oct 30 18:29:56.700648 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.700586 5232 scope.go:122] "RemoveContainer" containerID="b81d1ff926caddabf97a8403c2ef6a8a4fc4681483a8948773e9cf2e63574061" Oct 30 18:29:56.701069 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:29:56.701032 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b81d1ff926caddabf97a8403c2ef6a8a4fc4681483a8948773e9cf2e63574061\": container with ID starting with b81d1ff926caddabf97a8403c2ef6a8a4fc4681483a8948773e9cf2e63574061 not found: ID does not exist" containerID="b81d1ff926caddabf97a8403c2ef6a8a4fc4681483a8948773e9cf2e63574061" Oct 30 18:29:56.701162 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.701071 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b81d1ff926caddabf97a8403c2ef6a8a4fc4681483a8948773e9cf2e63574061"} err="failed to get container status \"b81d1ff926caddabf97a8403c2ef6a8a4fc4681483a8948773e9cf2e63574061\": rpc error: code = NotFound desc = could not find container \"b81d1ff926caddabf97a8403c2ef6a8a4fc4681483a8948773e9cf2e63574061\": container with ID starting with b81d1ff926caddabf97a8403c2ef6a8a4fc4681483a8948773e9cf2e63574061 not found: ID does not exist" Oct 30 18:29:56.701162 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.701097 5232 scope.go:122] "RemoveContainer" containerID="b0332b10b4ab6eb47cd6dccae366dc9a91d7bb2c384857f58eae8214ca10b7a0" Oct 30 18:29:56.711731 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.711709 5232 scope.go:122] "RemoveContainer" containerID="b0332b10b4ab6eb47cd6dccae366dc9a91d7bb2c384857f58eae8214ca10b7a0" Oct 30 18:29:56.712036 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:29:56.712016 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0332b10b4ab6eb47cd6dccae366dc9a91d7bb2c384857f58eae8214ca10b7a0\": container with ID starting with b0332b10b4ab6eb47cd6dccae366dc9a91d7bb2c384857f58eae8214ca10b7a0 not found: ID does not exist" containerID="b0332b10b4ab6eb47cd6dccae366dc9a91d7bb2c384857f58eae8214ca10b7a0" Oct 30 18:29:56.712103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.712044 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0332b10b4ab6eb47cd6dccae366dc9a91d7bb2c384857f58eae8214ca10b7a0"} err="failed to get container status \"b0332b10b4ab6eb47cd6dccae366dc9a91d7bb2c384857f58eae8214ca10b7a0\": rpc error: code = NotFound desc = could not find container \"b0332b10b4ab6eb47cd6dccae366dc9a91d7bb2c384857f58eae8214ca10b7a0\": container with ID starting with b0332b10b4ab6eb47cd6dccae366dc9a91d7bb2c384857f58eae8214ca10b7a0 not found: ID does not exist" Oct 30 18:29:56.712103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.712061 5232 scope.go:122] "RemoveContainer" containerID="8aeabf78f92ea63322770ddcba8644e2bc0939d47e9ce78e94c579eae24513ec" Oct 30 18:29:56.721599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.721574 5232 scope.go:122] "RemoveContainer" containerID="8aeabf78f92ea63322770ddcba8644e2bc0939d47e9ce78e94c579eae24513ec" Oct 30 18:29:56.721877 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:29:56.721859 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8aeabf78f92ea63322770ddcba8644e2bc0939d47e9ce78e94c579eae24513ec\": container with ID starting with 8aeabf78f92ea63322770ddcba8644e2bc0939d47e9ce78e94c579eae24513ec not found: ID does not exist" containerID="8aeabf78f92ea63322770ddcba8644e2bc0939d47e9ce78e94c579eae24513ec" Oct 30 18:29:56.721930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.721888 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8aeabf78f92ea63322770ddcba8644e2bc0939d47e9ce78e94c579eae24513ec"} err="failed to get container status \"8aeabf78f92ea63322770ddcba8644e2bc0939d47e9ce78e94c579eae24513ec\": rpc error: code = NotFound desc = could not find container \"8aeabf78f92ea63322770ddcba8644e2bc0939d47e9ce78e94c579eae24513ec\": container with ID starting with 8aeabf78f92ea63322770ddcba8644e2bc0939d47e9ce78e94c579eae24513ec not found: ID does not exist" Oct 30 18:29:56.739466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.739441 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k"] Oct 30 18:29:56.751357 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.751333 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k"] Oct 30 18:29:56.786786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.786754 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz"] Oct 30 18:29:56.794354 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.794330 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/distinct-w3-8b6cf1-kserve-58946f4c46-ksftz"] Oct 30 18:29:56.803824 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.803799 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" path="/var/lib/kubelet/pods/1a0e1051-f784-4141-a8fa-c5fb432f4abd/volumes" Oct 30 18:29:56.804414 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.804395 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" path="/var/lib/kubelet/pods/1ade2fbb-e0fe-4957-a7ff-a74990256474/volumes" Oct 30 18:29:56.847031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.846998 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz"] Oct 30 18:29:56.851087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:56.851060 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-8b6cf1/distinct-2-w3-8b6cf1-kserve-7bfb56f545-dbzhz"] Oct 30 18:29:57.026572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:57.015795 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-8b6cf1/premium-w3-8b6cf1-kserve-6d7647cbb7-qfv2k" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" probeResult="failure" output="Get \"https://10.133.0.48:8000/ready\": context deadline exceeded" Oct 30 18:29:57.075792 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:29:57.075758 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:29:58.803197 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:29:58.803160 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" path="/var/lib/kubelet/pods/e70bbbb7-6e09-46cf-9119-868fc52bb412/volumes" Oct 30 18:30:00.030680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.028250 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" Oct 30 18:30:00.119837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.119810 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" Oct 30 18:30:00.126344 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.126320 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc97fed8-9a76-4696-ab34-5a4a1d401ac3-kube-api-access-6nt68\" (UniqueName: \"kubernetes.io/projected/fc97fed8-9a76-4696-ab34-5a4a1d401ac3-kube-api-access-6nt68\") pod \"fc97fed8-9a76-4696-ab34-5a4a1d401ac3\" (UID: \"fc97fed8-9a76-4696-ab34-5a4a1d401ac3\") " Oct 30 18:30:00.126468 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.126414 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc97fed8-9a76-4696-ab34-5a4a1d401ac3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc97fed8-9a76-4696-ab34-5a4a1d401ac3-plugins-config-volume\") pod \"fc97fed8-9a76-4696-ab34-5a4a1d401ac3\" (UID: \"fc97fed8-9a76-4696-ab34-5a4a1d401ac3\") " Oct 30 18:30:00.126854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.126832 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc97fed8-9a76-4696-ab34-5a4a1d401ac3-plugins-config-volume" pod "fc97fed8-9a76-4696-ab34-5a4a1d401ac3" (UID: "fc97fed8-9a76-4696-ab34-5a4a1d401ac3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:00.127077 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.127062 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc97fed8-9a76-4696-ab34-5a4a1d401ac3-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:00.128610 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.128591 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc97fed8-9a76-4696-ab34-5a4a1d401ac3-kube-api-access-6nt68" pod "fc97fed8-9a76-4696-ab34-5a4a1d401ac3" (UID: "fc97fed8-9a76-4696-ab34-5a4a1d401ac3"). InnerVolumeSpecName "kube-api-access-6nt68". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:00.156708 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.156673 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv"] Oct 30 18:30:00.162017 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.161986 5232 cpu_manager.go:414] "containerMap: removing container" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" Oct 30 18:30:00.162017 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162015 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" Oct 30 18:30:00.162272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162042 5232 cpu_manager.go:414] "containerMap: removing container" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" Oct 30 18:30:00.162272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162051 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" Oct 30 18:30:00.162272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162067 5232 cpu_manager.go:414] "containerMap: removing container" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" Oct 30 18:30:00.162272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162075 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" Oct 30 18:30:00.162272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162099 5232 cpu_manager.go:414] "containerMap: removing container" podUID="476a0b93-3e62-4ae7-8287-4f620e7e016b" containerName="revoke-keys" Oct 30 18:30:00.162272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162106 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="476a0b93-3e62-4ae7-8287-4f620e7e016b" containerName="revoke-keys" Oct 30 18:30:00.162272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162123 5232 cpu_manager.go:414] "containerMap: removing container" podUID="9433bec1-83df-429a-98a3-4b573d4fb5f6" containerName="payload-pre-processing" Oct 30 18:30:00.162272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162130 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9433bec1-83df-429a-98a3-4b573d4fb5f6" containerName="payload-pre-processing" Oct 30 18:30:00.162272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162151 5232 cpu_manager.go:414] "containerMap: removing container" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" Oct 30 18:30:00.162272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162158 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" Oct 30 18:30:00.162272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162202 5232 cpu_manager.go:414] "containerMap: removing container" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" Oct 30 18:30:00.162272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162208 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" Oct 30 18:30:00.162272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162221 5232 cpu_manager.go:414] "containerMap: removing container" podUID="5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0" containerName="revoke-keys" Oct 30 18:30:00.162272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162228 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0" containerName="revoke-keys" Oct 30 18:30:00.162272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162246 5232 cpu_manager.go:414] "containerMap: removing container" podUID="fc97fed8-9a76-4696-ab34-5a4a1d401ac3" containerName="payload-processing" Oct 30 18:30:00.162272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162253 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc97fed8-9a76-4696-ab34-5a4a1d401ac3" containerName="payload-processing" Oct 30 18:30:00.163151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162384 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="9433bec1-83df-429a-98a3-4b573d4fb5f6" containerName="payload-pre-processing" Oct 30 18:30:00.163151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162410 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ade2fbb-e0fe-4957-a7ff-a74990256474" containerName="main" Oct 30 18:30:00.163151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162431 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="d74ee1cf-7d30-49b6-880d-a7368794823d" containerName="main" Oct 30 18:30:00.163151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162441 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0" containerName="revoke-keys" Oct 30 18:30:00.163151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162459 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc97fed8-9a76-4696-ab34-5a4a1d401ac3" containerName="payload-processing" Oct 30 18:30:00.163151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162470 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="05a41b45-1068-426b-bbf8-422e1b4980ba" containerName="main" Oct 30 18:30:00.163151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162480 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="e70bbbb7-6e09-46cf-9119-868fc52bb412" containerName="main" Oct 30 18:30:00.163151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162490 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a0e1051-f784-4141-a8fa-c5fb432f4abd" containerName="main" Oct 30 18:30:00.163151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.162500 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="476a0b93-3e62-4ae7-8287-4f620e7e016b" containerName="revoke-keys" Oct 30 18:30:00.165362 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.165338 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t"] Oct 30 18:30:00.165482 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.165457 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv" Oct 30 18:30:00.168709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.168688 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh"] Oct 30 18:30:00.168832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.168812 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" Oct 30 18:30:00.173277 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.173237 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" Oct 30 18:30:00.185262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.185238 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:30:00.187296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.187001 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv"] Oct 30 18:30:00.203789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.192800 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8"] Oct 30 18:30:00.203789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.199268 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t"] Oct 30 18:30:00.203789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.199389 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" Oct 30 18:30:00.203789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.202391 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd"] Oct 30 18:30:00.206008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.205927 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" Oct 30 18:30:00.221915 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.220587 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8"] Oct 30 18:30:00.228544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.228474 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9433bec1-83df-429a-98a3-4b573d4fb5f6-kube-api-access-h8r54\" (UniqueName: \"kubernetes.io/projected/9433bec1-83df-429a-98a3-4b573d4fb5f6-kube-api-access-h8r54\") pod \"9433bec1-83df-429a-98a3-4b573d4fb5f6\" (UID: \"9433bec1-83df-429a-98a3-4b573d4fb5f6\") " Oct 30 18:30:00.228781 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.228747 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9433bec1-83df-429a-98a3-4b573d4fb5f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9433bec1-83df-429a-98a3-4b573d4fb5f6-plugins-config-volume\") pod \"9433bec1-83df-429a-98a3-4b573d4fb5f6\" (UID: \"9433bec1-83df-429a-98a3-4b573d4fb5f6\") " Oct 30 18:30:00.229388 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.229366 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nt68\" (UniqueName: \"kubernetes.io/projected/fc97fed8-9a76-4696-ab34-5a4a1d401ac3-kube-api-access-6nt68\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:00.229828 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.229791 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9433bec1-83df-429a-98a3-4b573d4fb5f6-plugins-config-volume" pod "9433bec1-83df-429a-98a3-4b573d4fb5f6" (UID: "9433bec1-83df-429a-98a3-4b573d4fb5f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:00.230862 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.230839 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh"] Oct 30 18:30:00.242737 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.242713 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9433bec1-83df-429a-98a3-4b573d4fb5f6-kube-api-access-h8r54" pod "9433bec1-83df-429a-98a3-4b573d4fb5f6" (UID: "9433bec1-83df-429a-98a3-4b573d4fb5f6"). InnerVolumeSpecName "kube-api-access-h8r54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:00.243309 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.243285 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd"] Oct 30 18:30:00.256591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.256564 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22"] Oct 30 18:30:00.260099 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.260077 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22" Oct 30 18:30:00.280248 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.280210 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22"] Oct 30 18:30:00.283167 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.283139 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:30:00.330489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.330446 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pqjnj\" (UniqueName: \"kubernetes.io/projected/d43e57d7-9c37-45a0-9389-f52d40d839e2-kube-api-access-pqjnj\") pod \"maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t\" (UID: \"d43e57d7-9c37-45a0-9389-f52d40d839e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" Oct 30 18:30:00.330682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.330654 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca70bee7-f958-4bb5-9943-f6d3f721ed21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd\" (UID: \"ca70bee7-f958-4bb5-9943-f6d3f721ed21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" Oct 30 18:30:00.330728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.330709 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e342c0b-78c7-46f5-abcf-850309d6b6cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8\" (UID: \"5e342c0b-78c7-46f5-abcf-850309d6b6cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" Oct 30 18:30:00.330816 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.330792 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43e57d7-9c37-45a0-9389-f52d40d839e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t\" (UID: \"d43e57d7-9c37-45a0-9389-f52d40d839e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" Oct 30 18:30:00.330875 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.330860 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/851acf02-f2a8-456f-a8a6-4cba6df59cb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh\" (UID: \"851acf02-f2a8-456f-a8a6-4cba6df59cb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" Oct 30 18:30:00.331005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.330980 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13df76b-49f4-4942-aab5-7b0bf91ffad1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv\" (UID: \"f13df76b-49f4-4942-aab5-7b0bf91ffad1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv" Oct 30 18:30:00.331051 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.331035 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2zlv\" (UniqueName: \"kubernetes.io/projected/f13df76b-49f4-4942-aab5-7b0bf91ffad1-kube-api-access-n2zlv\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv\" (UID: \"f13df76b-49f4-4942-aab5-7b0bf91ffad1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv" Oct 30 18:30:00.331127 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.331105 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vnlf4\" (UniqueName: \"kubernetes.io/projected/ca70bee7-f958-4bb5-9943-f6d3f721ed21-kube-api-access-vnlf4\") pod \"maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd\" (UID: \"ca70bee7-f958-4bb5-9943-f6d3f721ed21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" Oct 30 18:30:00.331240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.331213 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dl7w9\" (UniqueName: \"kubernetes.io/projected/5e342c0b-78c7-46f5-abcf-850309d6b6cd-kube-api-access-dl7w9\") pod \"maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8\" (UID: \"5e342c0b-78c7-46f5-abcf-850309d6b6cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" Oct 30 18:30:00.331302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.331270 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d68t4\" (UniqueName: \"kubernetes.io/projected/851acf02-f2a8-456f-a8a6-4cba6df59cb2-kube-api-access-d68t4\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh\" (UID: \"851acf02-f2a8-456f-a8a6-4cba6df59cb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" Oct 30 18:30:00.331435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.331421 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9433bec1-83df-429a-98a3-4b573d4fb5f6-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:00.331592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.331442 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8r54\" (UniqueName: \"kubernetes.io/projected/9433bec1-83df-429a-98a3-4b573d4fb5f6-kube-api-access-h8r54\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:00.432474 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.432373 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2zlv\" (UniqueName: \"kubernetes.io/projected/f13df76b-49f4-4942-aab5-7b0bf91ffad1-kube-api-access-n2zlv\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv\" (UID: \"f13df76b-49f4-4942-aab5-7b0bf91ffad1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv" Oct 30 18:30:00.432474 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.432432 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vnlf4\" (UniqueName: \"kubernetes.io/projected/ca70bee7-f958-4bb5-9943-f6d3f721ed21-kube-api-access-vnlf4\") pod \"maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd\" (UID: \"ca70bee7-f958-4bb5-9943-f6d3f721ed21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" Oct 30 18:30:00.432735 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.432493 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dl7w9\" (UniqueName: \"kubernetes.io/projected/5e342c0b-78c7-46f5-abcf-850309d6b6cd-kube-api-access-dl7w9\") pod \"maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8\" (UID: \"5e342c0b-78c7-46f5-abcf-850309d6b6cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" Oct 30 18:30:00.432735 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.432566 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d68t4\" (UniqueName: \"kubernetes.io/projected/851acf02-f2a8-456f-a8a6-4cba6df59cb2-kube-api-access-d68t4\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh\" (UID: \"851acf02-f2a8-456f-a8a6-4cba6df59cb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" Oct 30 18:30:00.432735 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.432615 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pqjnj\" (UniqueName: \"kubernetes.io/projected/d43e57d7-9c37-45a0-9389-f52d40d839e2-kube-api-access-pqjnj\") pod \"maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t\" (UID: \"d43e57d7-9c37-45a0-9389-f52d40d839e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" Oct 30 18:30:00.432735 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.432665 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/391ee642-1ab6-47cc-b103-f381ea9fd5bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fbc0906-dvr22\" (UID: \"391ee642-1ab6-47cc-b103-f381ea9fd5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22" Oct 30 18:30:00.432735 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.432684 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/391ee642-1ab6-47cc-b103-f381ea9fd5bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fbc0906-dvr22\" (UID: \"391ee642-1ab6-47cc-b103-f381ea9fd5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22" Oct 30 18:30:00.432914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.432755 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca70bee7-f958-4bb5-9943-f6d3f721ed21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd\" (UID: \"ca70bee7-f958-4bb5-9943-f6d3f721ed21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" Oct 30 18:30:00.432914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.432803 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e342c0b-78c7-46f5-abcf-850309d6b6cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8\" (UID: \"5e342c0b-78c7-46f5-abcf-850309d6b6cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" Oct 30 18:30:00.432914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.432830 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43e57d7-9c37-45a0-9389-f52d40d839e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t\" (UID: \"d43e57d7-9c37-45a0-9389-f52d40d839e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" Oct 30 18:30:00.432914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.432872 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5blh\" (UniqueName: \"kubernetes.io/projected/391ee642-1ab6-47cc-b103-f381ea9fd5bd-kube-api-access-r5blh\") pod \"maas-api-revoke-sub-6fbc0906-dvr22\" (UID: \"391ee642-1ab6-47cc-b103-f381ea9fd5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22" Oct 30 18:30:00.432914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.432891 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/851acf02-f2a8-456f-a8a6-4cba6df59cb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh\" (UID: \"851acf02-f2a8-456f-a8a6-4cba6df59cb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" Oct 30 18:30:00.433082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.432939 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13df76b-49f4-4942-aab5-7b0bf91ffad1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv\" (UID: \"f13df76b-49f4-4942-aab5-7b0bf91ffad1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv" Oct 30 18:30:00.433704 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.433669 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e342c0b-78c7-46f5-abcf-850309d6b6cd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8\" (UID: \"5e342c0b-78c7-46f5-abcf-850309d6b6cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" Oct 30 18:30:00.433891 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.433669 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca70bee7-f958-4bb5-9943-f6d3f721ed21-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd\" (UID: \"ca70bee7-f958-4bb5-9943-f6d3f721ed21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" Oct 30 18:30:00.433891 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.433680 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13df76b-49f4-4942-aab5-7b0bf91ffad1-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv\" (UID: \"f13df76b-49f4-4942-aab5-7b0bf91ffad1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv" Oct 30 18:30:00.434045 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.433981 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43e57d7-9c37-45a0-9389-f52d40d839e2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t\" (UID: \"d43e57d7-9c37-45a0-9389-f52d40d839e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" Oct 30 18:30:00.434357 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.434339 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/851acf02-f2a8-456f-a8a6-4cba6df59cb2-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh\" (UID: \"851acf02-f2a8-456f-a8a6-4cba6df59cb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" Oct 30 18:30:00.464427 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.464391 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vnlf4\" (UniqueName: \"kubernetes.io/projected/ca70bee7-f958-4bb5-9943-f6d3f721ed21-kube-api-access-vnlf4\") pod \"maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd\" (UID: \"ca70bee7-f958-4bb5-9943-f6d3f721ed21\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" Oct 30 18:30:00.464590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.464391 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2zlv\" (UniqueName: \"kubernetes.io/projected/f13df76b-49f4-4942-aab5-7b0bf91ffad1-kube-api-access-n2zlv\") pod \"maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv\" (UID: \"f13df76b-49f4-4942-aab5-7b0bf91ffad1\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv" Oct 30 18:30:00.464590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.464498 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dl7w9\" (UniqueName: \"kubernetes.io/projected/5e342c0b-78c7-46f5-abcf-850309d6b6cd-kube-api-access-dl7w9\") pod \"maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8\" (UID: \"5e342c0b-78c7-46f5-abcf-850309d6b6cd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" Oct 30 18:30:00.464743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.464726 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pqjnj\" (UniqueName: \"kubernetes.io/projected/d43e57d7-9c37-45a0-9389-f52d40d839e2-kube-api-access-pqjnj\") pod \"maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t\" (UID: \"d43e57d7-9c37-45a0-9389-f52d40d839e2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" Oct 30 18:30:00.464866 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.464850 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d68t4\" (UniqueName: \"kubernetes.io/projected/851acf02-f2a8-456f-a8a6-4cba6df59cb2-kube-api-access-d68t4\") pod \"maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh\" (UID: \"851acf02-f2a8-456f-a8a6-4cba6df59cb2\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" Oct 30 18:30:00.489889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.489862 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv" Oct 30 18:30:00.506935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.506911 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" Oct 30 18:30:00.533736 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.533697 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5blh\" (UniqueName: \"kubernetes.io/projected/391ee642-1ab6-47cc-b103-f381ea9fd5bd-kube-api-access-r5blh\") pod \"maas-api-revoke-sub-6fbc0906-dvr22\" (UID: \"391ee642-1ab6-47cc-b103-f381ea9fd5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22" Oct 30 18:30:00.533929 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.533774 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" Oct 30 18:30:00.534119 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.534095 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/391ee642-1ab6-47cc-b103-f381ea9fd5bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fbc0906-dvr22\" (UID: \"391ee642-1ab6-47cc-b103-f381ea9fd5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22" Oct 30 18:30:00.534195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.534132 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/391ee642-1ab6-47cc-b103-f381ea9fd5bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fbc0906-dvr22\" (UID: \"391ee642-1ab6-47cc-b103-f381ea9fd5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22" Oct 30 18:30:00.534872 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.534837 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/391ee642-1ab6-47cc-b103-f381ea9fd5bd-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6fbc0906-dvr22\" (UID: \"391ee642-1ab6-47cc-b103-f381ea9fd5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22" Oct 30 18:30:00.538772 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.538100 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/391ee642-1ab6-47cc-b103-f381ea9fd5bd-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6fbc0906-dvr22\" (UID: \"391ee642-1ab6-47cc-b103-f381ea9fd5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22" Oct 30 18:30:00.549827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.549763 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" Oct 30 18:30:00.556198 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.556163 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" Oct 30 18:30:00.560448 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.560417 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5blh\" (UniqueName: \"kubernetes.io/projected/391ee642-1ab6-47cc-b103-f381ea9fd5bd-kube-api-access-r5blh\") pod \"maas-api-revoke-sub-6fbc0906-dvr22\" (UID: \"391ee642-1ab6-47cc-b103-f381ea9fd5bd\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22" Oct 30 18:30:00.590129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.589872 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22" Oct 30 18:30:00.730836 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.730803 5232 generic.go:356] "Generic (PLEG): container finished" podID="fc97fed8-9a76-4696-ab34-5a4a1d401ac3" containerID="4a9a7043a7f3301f1aea78e41a0f5f2d48f5deb104e8c7651932d6ed2e319fff" exitCode=0 Oct 30 18:30:00.731040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.730883 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" Oct 30 18:30:00.731040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.730940 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" event={"ID":"fc97fed8-9a76-4696-ab34-5a4a1d401ac3","Type":"ContainerDied","Data":"4a9a7043a7f3301f1aea78e41a0f5f2d48f5deb104e8c7651932d6ed2e319fff"} Oct 30 18:30:00.731040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.731001 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d" event={"ID":"fc97fed8-9a76-4696-ab34-5a4a1d401ac3","Type":"ContainerDied","Data":"b58520dc75782a98a90090241314659687cfc7ff16a75047d229edc2f554a26c"} Oct 30 18:30:00.731040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.731023 5232 scope.go:122] "RemoveContainer" containerID="4a9a7043a7f3301f1aea78e41a0f5f2d48f5deb104e8c7651932d6ed2e319fff" Oct 30 18:30:00.732994 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.732855 5232 generic.go:356] "Generic (PLEG): container finished" podID="9433bec1-83df-429a-98a3-4b573d4fb5f6" containerID="c03ea0f6812a8a607cf8f7f10635cfc832de40a11e0d0d31c6d2fd51817355fd" exitCode=0 Oct 30 18:30:00.732994 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.732910 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" Oct 30 18:30:00.732994 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.732948 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" event={"ID":"9433bec1-83df-429a-98a3-4b573d4fb5f6","Type":"ContainerDied","Data":"c03ea0f6812a8a607cf8f7f10635cfc832de40a11e0d0d31c6d2fd51817355fd"} Oct 30 18:30:00.733209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.733003 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj" event={"ID":"9433bec1-83df-429a-98a3-4b573d4fb5f6","Type":"ContainerDied","Data":"c27595961475d25535d77368e1788bd45978ccba68a8f888bf9ef3329a4fa146"} Oct 30 18:30:00.753590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.744445 5232 scope.go:122] "RemoveContainer" containerID="4a9a7043a7f3301f1aea78e41a0f5f2d48f5deb104e8c7651932d6ed2e319fff" Oct 30 18:30:00.753590 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:00.744961 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a9a7043a7f3301f1aea78e41a0f5f2d48f5deb104e8c7651932d6ed2e319fff\": container with ID starting with 4a9a7043a7f3301f1aea78e41a0f5f2d48f5deb104e8c7651932d6ed2e319fff not found: ID does not exist" containerID="4a9a7043a7f3301f1aea78e41a0f5f2d48f5deb104e8c7651932d6ed2e319fff" Oct 30 18:30:00.753590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.744999 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a9a7043a7f3301f1aea78e41a0f5f2d48f5deb104e8c7651932d6ed2e319fff"} err="failed to get container status \"4a9a7043a7f3301f1aea78e41a0f5f2d48f5deb104e8c7651932d6ed2e319fff\": rpc error: code = NotFound desc = could not find container \"4a9a7043a7f3301f1aea78e41a0f5f2d48f5deb104e8c7651932d6ed2e319fff\": container with ID starting with 4a9a7043a7f3301f1aea78e41a0f5f2d48f5deb104e8c7651932d6ed2e319fff not found: ID does not exist" Oct 30 18:30:00.753590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.745022 5232 scope.go:122] "RemoveContainer" containerID="c03ea0f6812a8a607cf8f7f10635cfc832de40a11e0d0d31c6d2fd51817355fd" Oct 30 18:30:00.759007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.758985 5232 scope.go:122] "RemoveContainer" containerID="c03ea0f6812a8a607cf8f7f10635cfc832de40a11e0d0d31c6d2fd51817355fd" Oct 30 18:30:00.759382 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:00.759360 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c03ea0f6812a8a607cf8f7f10635cfc832de40a11e0d0d31c6d2fd51817355fd\": container with ID starting with c03ea0f6812a8a607cf8f7f10635cfc832de40a11e0d0d31c6d2fd51817355fd not found: ID does not exist" containerID="c03ea0f6812a8a607cf8f7f10635cfc832de40a11e0d0d31c6d2fd51817355fd" Oct 30 18:30:00.759474 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.759392 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c03ea0f6812a8a607cf8f7f10635cfc832de40a11e0d0d31c6d2fd51817355fd"} err="failed to get container status \"c03ea0f6812a8a607cf8f7f10635cfc832de40a11e0d0d31c6d2fd51817355fd\": rpc error: code = NotFound desc = could not find container \"c03ea0f6812a8a607cf8f7f10635cfc832de40a11e0d0d31c6d2fd51817355fd\": container with ID starting with c03ea0f6812a8a607cf8f7f10635cfc832de40a11e0d0d31c6d2fd51817355fd not found: ID does not exist" Oct 30 18:30:00.788330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.788294 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj"] Oct 30 18:30:00.788495 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.788341 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-fb2fa0-59bfcb55ff-sh4gj"] Oct 30 18:30:00.796840 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:30:00.796809 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf13df76b_49f4_4942_aab5_7b0bf91ffad1.slice/crio-da4b173755b172c55c46e67a7ac7239a197d1be3f20d354116ced7b1ed50222b WatchSource:0}: Error finding container da4b173755b172c55c46e67a7ac7239a197d1be3f20d354116ced7b1ed50222b: Status 404 returned error can't find the container with id da4b173755b172c55c46e67a7ac7239a197d1be3f20d354116ced7b1ed50222b Oct 30 18:30:00.814200 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.814162 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9433bec1-83df-429a-98a3-4b573d4fb5f6" path="/var/lib/kubelet/pods/9433bec1-83df-429a-98a3-4b573d4fb5f6/volumes" Oct 30 18:30:00.814684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.814650 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv"] Oct 30 18:30:00.824467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.824443 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t"] Oct 30 18:30:00.828690 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.828664 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d"] Oct 30 18:30:00.830257 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:30:00.830232 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd43e57d7_9c37_45a0_9389_f52d40d839e2.slice/crio-e3483382be43906bf8d39b9ae3ebb2f139ad4a1ab03efc0fa72d8d35d3784219 WatchSource:0}: Error finding container e3483382be43906bf8d39b9ae3ebb2f139ad4a1ab03efc0fa72d8d35d3784219: Status 404 returned error can't find the container with id e3483382be43906bf8d39b9ae3ebb2f139ad4a1ab03efc0fa72d8d35d3784219 Oct 30 18:30:00.842856 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.835877 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-fb2fa0-755dc5cf4-9w86d"] Oct 30 18:30:00.869796 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.869769 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh"] Oct 30 18:30:00.916310 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.916240 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd"] Oct 30 18:30:00.922095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.922074 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8"] Oct 30 18:30:00.942948 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:00.942924 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22"] Oct 30 18:30:01.015972 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:30:01.015928 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podca70bee7_f958_4bb5_9943_f6d3f721ed21.slice/crio-c808d32ad56d3c1236d2eab695bb648eaca20263d01261a7fba20e79a251d5a0 WatchSource:0}: Error finding container c808d32ad56d3c1236d2eab695bb648eaca20263d01261a7fba20e79a251d5a0: Status 404 returned error can't find the container with id c808d32ad56d3c1236d2eab695bb648eaca20263d01261a7fba20e79a251d5a0 Oct 30 18:30:01.039073 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.039036 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d"] Oct 30 18:30:01.044142 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.044115 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-19c62263-q8f7d"] Oct 30 18:30:01.114814 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:30:01.114781 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e342c0b_78c7_46f5_abcf_850309d6b6cd.slice/crio-203efaa47dadb6fa3bce3a3f3264f21766f3c2116df7faaac15cd681d91f7274 WatchSource:0}: Error finding container 203efaa47dadb6fa3bce3a3f3264f21766f3c2116df7faaac15cd681d91f7274: Status 404 returned error can't find the container with id 203efaa47dadb6fa3bce3a3f3264f21766f3c2116df7faaac15cd681d91f7274 Oct 30 18:30:01.115197 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:30:01.115171 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391ee642_1ab6_47cc_b103_f381ea9fd5bd.slice/crio-ca90d46986c6b46bea9a01f3d08315a2534dc9322427578bd164822f00dd01d4 WatchSource:0}: Error finding container ca90d46986c6b46bea9a01f3d08315a2534dc9322427578bd164822f00dd01d4: Status 404 returned error can't find the container with id ca90d46986c6b46bea9a01f3d08315a2534dc9322427578bd164822f00dd01d4 Oct 30 18:30:01.743208 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.743172 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" event={"ID":"851acf02-f2a8-456f-a8a6-4cba6df59cb2","Type":"ContainerStarted","Data":"253c0ebfbb697a9139781c5f980e3e9ef326bf1c13e17fbd499add2ee302cb97"} Oct 30 18:30:01.743372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.743240 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" event={"ID":"851acf02-f2a8-456f-a8a6-4cba6df59cb2","Type":"ContainerStarted","Data":"a1e059d9da3a8e49cdab0c17fb9d47a2bf3fcc4d5e95bc6c7f5fdae2d5fa6b6d"} Oct 30 18:30:01.746785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.746754 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" event={"ID":"ca70bee7-f958-4bb5-9943-f6d3f721ed21","Type":"ContainerStarted","Data":"826722caaa21aac1257d3c6cd0ae12f19d130023a2bc5da3efdc9bdd8c7f236e"} Oct 30 18:30:01.746947 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.746806 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" event={"ID":"ca70bee7-f958-4bb5-9943-f6d3f721ed21","Type":"ContainerStarted","Data":"c808d32ad56d3c1236d2eab695bb648eaca20263d01261a7fba20e79a251d5a0"} Oct 30 18:30:01.748540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.748484 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" event={"ID":"5e342c0b-78c7-46f5-abcf-850309d6b6cd","Type":"ContainerStarted","Data":"4e82c5a66ad3d6c70029459bc767f162423f3081830fa7dbc2024c09b86e3dad"} Oct 30 18:30:01.748668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.748559 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" event={"ID":"5e342c0b-78c7-46f5-abcf-850309d6b6cd","Type":"ContainerStarted","Data":"203efaa47dadb6fa3bce3a3f3264f21766f3c2116df7faaac15cd681d91f7274"} Oct 30 18:30:01.750336 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.750305 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" event={"ID":"d43e57d7-9c37-45a0-9389-f52d40d839e2","Type":"ContainerStarted","Data":"79005591e4c75f20b01745bedea109a8deefb1fc15b1a78b0157bedd33fbc6c8"} Oct 30 18:30:01.750437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.750359 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" event={"ID":"d43e57d7-9c37-45a0-9389-f52d40d839e2","Type":"ContainerStarted","Data":"e3483382be43906bf8d39b9ae3ebb2f139ad4a1ab03efc0fa72d8d35d3784219"} Oct 30 18:30:01.751867 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.751847 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv" event={"ID":"f13df76b-49f4-4942-aab5-7b0bf91ffad1","Type":"ContainerStarted","Data":"8f47b9d82cd8bcced5dede0a116dcb7904be1ca771cda5833ba93fb274b5e10b"} Oct 30 18:30:01.751955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.751885 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv" event={"ID":"f13df76b-49f4-4942-aab5-7b0bf91ffad1","Type":"ContainerStarted","Data":"da4b173755b172c55c46e67a7ac7239a197d1be3f20d354116ced7b1ed50222b"} Oct 30 18:30:01.753320 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.753298 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22" event={"ID":"391ee642-1ab6-47cc-b103-f381ea9fd5bd","Type":"ContainerStarted","Data":"d9e1f544f3a0e7de1723c765c08627973d4e4e0e245ed3c5054326a383218caa"} Oct 30 18:30:01.753401 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.753335 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22" event={"ID":"391ee642-1ab6-47cc-b103-f381ea9fd5bd","Type":"ContainerStarted","Data":"ca90d46986c6b46bea9a01f3d08315a2534dc9322427578bd164822f00dd01d4"} Oct 30 18:30:01.784317 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.784211 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" podStartSLOduration=1.7841948269999999 podStartE2EDuration="1.784194827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:01.782419005 +0000 UTC m=+1701.098839329" watchObservedRunningTime="2026-10-30 18:30:01.784194827 +0000 UTC m=+1701.100615147" Oct 30 18:30:01.822161 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.822110 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22" podStartSLOduration=1.8220959300000001 podStartE2EDuration="1.82209593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:01.820399103 +0000 UTC m=+1701.136819425" watchObservedRunningTime="2026-10-30 18:30:01.82209593 +0000 UTC m=+1701.138516251" Oct 30 18:30:01.862135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.862087 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv" podStartSLOduration=1.8620720099999999 podStartE2EDuration="1.86207201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:01.858585005 +0000 UTC m=+1701.175005328" watchObservedRunningTime="2026-10-30 18:30:01.86207201 +0000 UTC m=+1701.178492331" Oct 30 18:30:01.898558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.898462 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" podStartSLOduration=1.898450112 podStartE2EDuration="1.898450112s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:01.895344388 +0000 UTC m=+1701.211764710" watchObservedRunningTime="2026-10-30 18:30:01.898450112 +0000 UTC m=+1701.214870433" Oct 30 18:30:01.939987 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.939934 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" podStartSLOduration=1.9399191089999999 podStartE2EDuration="1.939919109s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:01.938686458 +0000 UTC m=+1701.255106790" watchObservedRunningTime="2026-10-30 18:30:01.939919109 +0000 UTC m=+1701.256339449" Oct 30 18:30:01.980822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:01.980758 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" podStartSLOduration=1.9807427990000002 podStartE2EDuration="1.980742799s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:01.978705139 +0000 UTC m=+1701.295125461" watchObservedRunningTime="2026-10-30 18:30:01.980742799 +0000 UTC m=+1701.297163114" Oct 30 18:30:02.048035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.047953 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9"] Oct 30 18:30:02.064066 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.062043 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2cf6fe24-jflc9"] Oct 30 18:30:02.105538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.105472 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv"] Oct 30 18:30:02.688964 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.688866 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk"] Oct 30 18:30:02.692001 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.691971 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-fb2fa0-bzjdk"] Oct 30 18:30:02.764840 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.764805 5232 generic.go:356] "Generic (PLEG): container finished" podID="ca70bee7-f958-4bb5-9943-f6d3f721ed21" containerID="826722caaa21aac1257d3c6cd0ae12f19d130023a2bc5da3efdc9bdd8c7f236e" exitCode=60 Oct 30 18:30:02.765039 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.764891 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" event={"ID":"ca70bee7-f958-4bb5-9943-f6d3f721ed21","Type":"ContainerDied","Data":"826722caaa21aac1257d3c6cd0ae12f19d130023a2bc5da3efdc9bdd8c7f236e"} Oct 30 18:30:02.765153 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.765135 5232 scope.go:122] "RemoveContainer" containerID="826722caaa21aac1257d3c6cd0ae12f19d130023a2bc5da3efdc9bdd8c7f236e" Oct 30 18:30:02.766763 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.766743 5232 generic.go:356] "Generic (PLEG): container finished" podID="5e342c0b-78c7-46f5-abcf-850309d6b6cd" containerID="4e82c5a66ad3d6c70029459bc767f162423f3081830fa7dbc2024c09b86e3dad" exitCode=60 Oct 30 18:30:02.766869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.766841 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" event={"ID":"5e342c0b-78c7-46f5-abcf-850309d6b6cd","Type":"ContainerDied","Data":"4e82c5a66ad3d6c70029459bc767f162423f3081830fa7dbc2024c09b86e3dad"} Oct 30 18:30:02.767056 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.767043 5232 scope.go:122] "RemoveContainer" containerID="4e82c5a66ad3d6c70029459bc767f162423f3081830fa7dbc2024c09b86e3dad" Oct 30 18:30:02.768842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.768703 5232 generic.go:356] "Generic (PLEG): container finished" podID="d43e57d7-9c37-45a0-9389-f52d40d839e2" containerID="79005591e4c75f20b01745bedea109a8deefb1fc15b1a78b0157bedd33fbc6c8" exitCode=60 Oct 30 18:30:02.768842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.768804 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" event={"ID":"d43e57d7-9c37-45a0-9389-f52d40d839e2","Type":"ContainerDied","Data":"79005591e4c75f20b01745bedea109a8deefb1fc15b1a78b0157bedd33fbc6c8"} Oct 30 18:30:02.769016 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.768999 5232 scope.go:122] "RemoveContainer" containerID="79005591e4c75f20b01745bedea109a8deefb1fc15b1a78b0157bedd33fbc6c8" Oct 30 18:30:02.770778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.770753 5232 generic.go:356] "Generic (PLEG): container finished" podID="f13df76b-49f4-4942-aab5-7b0bf91ffad1" containerID="8f47b9d82cd8bcced5dede0a116dcb7904be1ca771cda5833ba93fb274b5e10b" exitCode=60 Oct 30 18:30:02.770867 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.770797 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv" event={"ID":"f13df76b-49f4-4942-aab5-7b0bf91ffad1","Type":"ContainerDied","Data":"8f47b9d82cd8bcced5dede0a116dcb7904be1ca771cda5833ba93fb274b5e10b"} Oct 30 18:30:02.770981 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.770963 5232 scope.go:122] "RemoveContainer" containerID="8f47b9d82cd8bcced5dede0a116dcb7904be1ca771cda5833ba93fb274b5e10b" Oct 30 18:30:02.772673 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.772652 5232 generic.go:356] "Generic (PLEG): container finished" podID="391ee642-1ab6-47cc-b103-f381ea9fd5bd" containerID="d9e1f544f3a0e7de1723c765c08627973d4e4e0e245ed3c5054326a383218caa" exitCode=0 Oct 30 18:30:02.772761 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.772737 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22" event={"ID":"391ee642-1ab6-47cc-b103-f381ea9fd5bd","Type":"ContainerDied","Data":"d9e1f544f3a0e7de1723c765c08627973d4e4e0e245ed3c5054326a383218caa"} Oct 30 18:30:02.774406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.774388 5232 generic.go:356] "Generic (PLEG): container finished" podID="851acf02-f2a8-456f-a8a6-4cba6df59cb2" containerID="253c0ebfbb697a9139781c5f980e3e9ef326bf1c13e17fbd499add2ee302cb97" exitCode=60 Oct 30 18:30:02.774498 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.774475 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" event={"ID":"851acf02-f2a8-456f-a8a6-4cba6df59cb2","Type":"ContainerDied","Data":"253c0ebfbb697a9139781c5f980e3e9ef326bf1c13e17fbd499add2ee302cb97"} Oct 30 18:30:02.774775 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.774757 5232 scope.go:122] "RemoveContainer" containerID="253c0ebfbb697a9139781c5f980e3e9ef326bf1c13e17fbd499add2ee302cb97" Oct 30 18:30:02.813094 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.813017 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="482da54b-63ab-4595-9d44-3ca24ed32d03" path="/var/lib/kubelet/pods/482da54b-63ab-4595-9d44-3ca24ed32d03/volumes" Oct 30 18:30:02.813677 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.813653 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="546beae5-c98b-4b49-9bc8-46b77e64648c" path="/var/lib/kubelet/pods/546beae5-c98b-4b49-9bc8-46b77e64648c/volumes" Oct 30 18:30:02.814224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.814208 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82285c8d-b803-4f2d-9ca7-ef302cf7b5fc" path="/var/lib/kubelet/pods/82285c8d-b803-4f2d-9ca7-ef302cf7b5fc/volumes" Oct 30 18:30:02.816332 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:02.816308 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc97fed8-9a76-4696-ab34-5a4a1d401ac3" path="/var/lib/kubelet/pods/fc97fed8-9a76-4696-ab34-5a4a1d401ac3/volumes" Oct 30 18:30:03.682482 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.682402 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv"] Oct 30 18:30:03.685637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.685614 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:03.701716 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.701688 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv"] Oct 30 18:30:03.715464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.710460 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1b0848\"/\"openshift-service-ca.crt\"" Oct 30 18:30:03.715464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.710483 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-1b0848\"/\"kube-root-ca.crt\"" Oct 30 18:30:03.715464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.710612 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1b0848\"/\"default-dockercfg-kmrf2\"" Oct 30 18:30:03.715464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.710645 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-1b0848\"/\"test-model-1b0848-kserve-self-signed-certs\"" Oct 30 18:30:03.776859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.776823 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-model-cache\") pod \"test-model-1b0848-kserve-648bb66d45-xf6tv\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:03.777039 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.776900 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-tmp-dir\") pod \"test-model-1b0848-kserve-648bb66d45-xf6tv\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:03.777039 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.776954 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfcjp\" (UniqueName: \"kubernetes.io/projected/3a723eea-87a6-48d9-b2cd-a2df079841c6-kube-api-access-hfcjp\") pod \"test-model-1b0848-kserve-648bb66d45-xf6tv\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:03.777039 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.776993 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-home\") pod \"test-model-1b0848-kserve-648bb66d45-xf6tv\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:03.777257 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.777236 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a723eea-87a6-48d9-b2cd-a2df079841c6-tls-certs\") pod \"test-model-1b0848-kserve-648bb66d45-xf6tv\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:03.777354 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.777335 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-dshm\") pod \"test-model-1b0848-kserve-648bb66d45-xf6tv\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:03.786053 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.786023 5232 generic.go:356] "Generic (PLEG): container finished" podID="851acf02-f2a8-456f-a8a6-4cba6df59cb2" containerID="1a1aee155423924283ee8bdc8e3bfcbf569b33fe81baae020f2741815035444a" exitCode=60 Oct 30 18:30:03.786199 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.786103 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" event={"ID":"851acf02-f2a8-456f-a8a6-4cba6df59cb2","Type":"ContainerDied","Data":"1a1aee155423924283ee8bdc8e3bfcbf569b33fe81baae020f2741815035444a"} Oct 30 18:30:03.786199 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.786166 5232 scope.go:122] "RemoveContainer" containerID="253c0ebfbb697a9139781c5f980e3e9ef326bf1c13e17fbd499add2ee302cb97" Oct 30 18:30:03.786586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.786553 5232 scope.go:122] "RemoveContainer" containerID="1a1aee155423924283ee8bdc8e3bfcbf569b33fe81baae020f2741815035444a" Oct 30 18:30:03.786875 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:03.786845 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh_odh-ai-gateway-infra(851acf02-f2a8-456f-a8a6-4cba6df59cb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" podUID="851acf02-f2a8-456f-a8a6-4cba6df59cb2" Oct 30 18:30:03.789110 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.789083 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" event={"ID":"ca70bee7-f958-4bb5-9943-f6d3f721ed21","Type":"ContainerStarted","Data":"aab66aca7ce72747bf780fd6932946661dff2c3c7bddb113a98de8c4db0b76ad"} Oct 30 18:30:03.791166 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.791138 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" event={"ID":"5e342c0b-78c7-46f5-abcf-850309d6b6cd","Type":"ContainerStarted","Data":"df2da92e73bb9ae809523c1cfbbcb5859268b634d33e330ad16c1fb230232a0c"} Oct 30 18:30:03.793381 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.793358 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" event={"ID":"d43e57d7-9c37-45a0-9389-f52d40d839e2","Type":"ContainerStarted","Data":"7df169fd24652a771524e93cfe3dc294a70d66550f3b055ac1e2911ed1249faf"} Oct 30 18:30:03.795757 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.795735 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv" event={"ID":"f13df76b-49f4-4942-aab5-7b0bf91ffad1","Type":"ContainerStarted","Data":"3424559a499b8a9549a854f707792468ee32242b348e359e9c8ccfc77c2844a1"} Oct 30 18:30:03.795859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.795792 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv" podUID="f13df76b-49f4-4942-aab5-7b0bf91ffad1" containerName="cleanup" containerID="cri-o://3424559a499b8a9549a854f707792468ee32242b348e359e9c8ccfc77c2844a1" gracePeriod=30 Oct 30 18:30:03.883238 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.882049 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-model-cache\") pod \"test-model-1b0848-kserve-648bb66d45-xf6tv\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:03.883238 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.882100 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-tmp-dir\") pod \"test-model-1b0848-kserve-648bb66d45-xf6tv\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:03.883238 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.882182 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfcjp\" (UniqueName: \"kubernetes.io/projected/3a723eea-87a6-48d9-b2cd-a2df079841c6-kube-api-access-hfcjp\") pod \"test-model-1b0848-kserve-648bb66d45-xf6tv\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:03.883238 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.882243 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-home\") pod \"test-model-1b0848-kserve-648bb66d45-xf6tv\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:03.883238 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.882464 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-model-cache\") pod \"test-model-1b0848-kserve-648bb66d45-xf6tv\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:03.883238 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.882716 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-tmp-dir\") pod \"test-model-1b0848-kserve-648bb66d45-xf6tv\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:03.883238 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.882746 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-home\") pod \"test-model-1b0848-kserve-648bb66d45-xf6tv\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:03.883238 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.882858 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a723eea-87a6-48d9-b2cd-a2df079841c6-tls-certs\") pod \"test-model-1b0848-kserve-648bb66d45-xf6tv\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:03.885241 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.885144 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-dshm\") pod \"test-model-1b0848-kserve-648bb66d45-xf6tv\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:03.888428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.887817 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a723eea-87a6-48d9-b2cd-a2df079841c6-tls-certs\") pod \"test-model-1b0848-kserve-648bb66d45-xf6tv\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:03.888655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.888633 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-dshm\") pod \"test-model-1b0848-kserve-648bb66d45-xf6tv\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:03.910927 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:03.910892 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfcjp\" (UniqueName: \"kubernetes.io/projected/3a723eea-87a6-48d9-b2cd-a2df079841c6-kube-api-access-hfcjp\") pod \"test-model-1b0848-kserve-648bb66d45-xf6tv\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:04.026549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.019183 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:04.117378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.089404 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22" Oct 30 18:30:04.194379 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.193343 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/391ee642-1ab6-47cc-b103-f381ea9fd5bd-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/391ee642-1ab6-47cc-b103-f381ea9fd5bd-maas-api-openshift-ca\") pod \"391ee642-1ab6-47cc-b103-f381ea9fd5bd\" (UID: \"391ee642-1ab6-47cc-b103-f381ea9fd5bd\") " Oct 30 18:30:04.194379 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.193450 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/391ee642-1ab6-47cc-b103-f381ea9fd5bd-kube-api-access-r5blh\" (UniqueName: \"kubernetes.io/projected/391ee642-1ab6-47cc-b103-f381ea9fd5bd-kube-api-access-r5blh\") pod \"391ee642-1ab6-47cc-b103-f381ea9fd5bd\" (UID: \"391ee642-1ab6-47cc-b103-f381ea9fd5bd\") " Oct 30 18:30:04.194379 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.193565 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/391ee642-1ab6-47cc-b103-f381ea9fd5bd-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/391ee642-1ab6-47cc-b103-f381ea9fd5bd-maas-api-xks-ca\") pod \"391ee642-1ab6-47cc-b103-f381ea9fd5bd\" (UID: \"391ee642-1ab6-47cc-b103-f381ea9fd5bd\") " Oct 30 18:30:04.195360 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.195318 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391ee642-1ab6-47cc-b103-f381ea9fd5bd-maas-api-openshift-ca" pod "391ee642-1ab6-47cc-b103-f381ea9fd5bd" (UID: "391ee642-1ab6-47cc-b103-f381ea9fd5bd"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:04.200316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.200266 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/391ee642-1ab6-47cc-b103-f381ea9fd5bd-maas-api-xks-ca" pod "391ee642-1ab6-47cc-b103-f381ea9fd5bd" (UID: "391ee642-1ab6-47cc-b103-f381ea9fd5bd"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:30:04.201804 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.201774 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/391ee642-1ab6-47cc-b103-f381ea9fd5bd-kube-api-access-r5blh" pod "391ee642-1ab6-47cc-b103-f381ea9fd5bd" (UID: "391ee642-1ab6-47cc-b103-f381ea9fd5bd"). InnerVolumeSpecName "kube-api-access-r5blh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:04.295203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.295164 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/391ee642-1ab6-47cc-b103-f381ea9fd5bd-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:04.295203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.295198 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5blh\" (UniqueName: \"kubernetes.io/projected/391ee642-1ab6-47cc-b103-f381ea9fd5bd-kube-api-access-r5blh\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:04.295203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.295210 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/391ee642-1ab6-47cc-b103-f381ea9fd5bd-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:04.430611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.430577 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv"] Oct 30 18:30:04.805460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.805422 5232 generic.go:356] "Generic (PLEG): container finished" podID="d43e57d7-9c37-45a0-9389-f52d40d839e2" containerID="7df169fd24652a771524e93cfe3dc294a70d66550f3b055ac1e2911ed1249faf" exitCode=60 Oct 30 18:30:04.806001 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.805502 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" event={"ID":"d43e57d7-9c37-45a0-9389-f52d40d839e2","Type":"ContainerDied","Data":"7df169fd24652a771524e93cfe3dc294a70d66550f3b055ac1e2911ed1249faf"} Oct 30 18:30:04.806001 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.805573 5232 scope.go:122] "RemoveContainer" containerID="79005591e4c75f20b01745bedea109a8deefb1fc15b1a78b0157bedd33fbc6c8" Oct 30 18:30:04.806001 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.805765 5232 scope.go:122] "RemoveContainer" containerID="7df169fd24652a771524e93cfe3dc294a70d66550f3b055ac1e2911ed1249faf" Oct 30 18:30:04.806174 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:04.806074 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t_odh-ai-gateway-infra(d43e57d7-9c37-45a0-9389-f52d40d839e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" podUID="d43e57d7-9c37-45a0-9389-f52d40d839e2" Oct 30 18:30:04.807054 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.807037 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22" Oct 30 18:30:04.807143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.807070 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22" event={"ID":"391ee642-1ab6-47cc-b103-f381ea9fd5bd","Type":"ContainerDied","Data":"ca90d46986c6b46bea9a01f3d08315a2534dc9322427578bd164822f00dd01d4"} Oct 30 18:30:04.807143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.807114 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca90d46986c6b46bea9a01f3d08315a2534dc9322427578bd164822f00dd01d4" Oct 30 18:30:04.809155 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.809135 5232 scope.go:122] "RemoveContainer" containerID="1a1aee155423924283ee8bdc8e3bfcbf569b33fe81baae020f2741815035444a" Oct 30 18:30:04.809409 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:04.809382 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh_odh-ai-gateway-infra(851acf02-f2a8-456f-a8a6-4cba6df59cb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" podUID="851acf02-f2a8-456f-a8a6-4cba6df59cb2" Oct 30 18:30:04.811237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.811214 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" event={"ID":"3a723eea-87a6-48d9-b2cd-a2df079841c6","Type":"ContainerStarted","Data":"330ac5a549fdf4842a4923e935c72fe05c20de44e3b3f3d980f5d3af5aeae5b3"} Oct 30 18:30:04.811345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.811245 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" event={"ID":"3a723eea-87a6-48d9-b2cd-a2df079841c6","Type":"ContainerStarted","Data":"8b7bc602141cb4c91ab0130a4884ce514f2447ba0be07b487df3ece67de7de69"} Oct 30 18:30:04.813473 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.813448 5232 generic.go:356] "Generic (PLEG): container finished" podID="ca70bee7-f958-4bb5-9943-f6d3f721ed21" containerID="aab66aca7ce72747bf780fd6932946661dff2c3c7bddb113a98de8c4db0b76ad" exitCode=60 Oct 30 18:30:04.813582 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.813557 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" event={"ID":"ca70bee7-f958-4bb5-9943-f6d3f721ed21","Type":"ContainerDied","Data":"aab66aca7ce72747bf780fd6932946661dff2c3c7bddb113a98de8c4db0b76ad"} Oct 30 18:30:04.813864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.813728 5232 scope.go:122] "RemoveContainer" containerID="aab66aca7ce72747bf780fd6932946661dff2c3c7bddb113a98de8c4db0b76ad" Oct 30 18:30:04.813970 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:04.813950 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd_odh-ai-gateway-infra(ca70bee7-f958-4bb5-9943-f6d3f721ed21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" podUID="ca70bee7-f958-4bb5-9943-f6d3f721ed21" Oct 30 18:30:04.815791 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.815765 5232 generic.go:356] "Generic (PLEG): container finished" podID="5e342c0b-78c7-46f5-abcf-850309d6b6cd" containerID="df2da92e73bb9ae809523c1cfbbcb5859268b634d33e330ad16c1fb230232a0c" exitCode=60 Oct 30 18:30:04.815885 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.815807 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" event={"ID":"5e342c0b-78c7-46f5-abcf-850309d6b6cd","Type":"ContainerDied","Data":"df2da92e73bb9ae809523c1cfbbcb5859268b634d33e330ad16c1fb230232a0c"} Oct 30 18:30:04.816037 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.816018 5232 scope.go:122] "RemoveContainer" containerID="df2da92e73bb9ae809523c1cfbbcb5859268b634d33e330ad16c1fb230232a0c" Oct 30 18:30:04.816233 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:04.816211 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8_odh-ai-gateway-infra(5e342c0b-78c7-46f5-abcf-850309d6b6cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" podUID="5e342c0b-78c7-46f5-abcf-850309d6b6cd" Oct 30 18:30:04.819005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.818989 5232 scope.go:122] "RemoveContainer" containerID="826722caaa21aac1257d3c6cd0ae12f19d130023a2bc5da3efdc9bdd8c7f236e" Oct 30 18:30:04.831021 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.831001 5232 scope.go:122] "RemoveContainer" containerID="4e82c5a66ad3d6c70029459bc767f162423f3081830fa7dbc2024c09b86e3dad" Oct 30 18:30:04.889592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:04.889540 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" podStartSLOduration=1.88949996 podStartE2EDuration="1.88949996s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:30:04.888337877 +0000 UTC m=+1704.204758235" watchObservedRunningTime="2026-10-30 18:30:04.88949996 +0000 UTC m=+1704.205920282" Oct 30 18:30:05.032531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:05.026558 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:05.825337 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:05.825309 5232 scope.go:122] "RemoveContainer" containerID="aab66aca7ce72747bf780fd6932946661dff2c3c7bddb113a98de8c4db0b76ad" Oct 30 18:30:05.825813 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:05.825701 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd_odh-ai-gateway-infra(ca70bee7-f958-4bb5-9943-f6d3f721ed21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" podUID="ca70bee7-f958-4bb5-9943-f6d3f721ed21" Oct 30 18:30:05.827065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:05.827044 5232 scope.go:122] "RemoveContainer" containerID="df2da92e73bb9ae809523c1cfbbcb5859268b634d33e330ad16c1fb230232a0c" Oct 30 18:30:05.827330 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:05.827309 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8_odh-ai-gateway-infra(5e342c0b-78c7-46f5-abcf-850309d6b6cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" podUID="5e342c0b-78c7-46f5-abcf-850309d6b6cd" Oct 30 18:30:05.828950 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:05.828934 5232 scope.go:122] "RemoveContainer" containerID="7df169fd24652a771524e93cfe3dc294a70d66550f3b055ac1e2911ed1249faf" Oct 30 18:30:05.829121 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:05.829104 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t_odh-ai-gateway-infra(d43e57d7-9c37-45a0-9389-f52d40d839e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" podUID="d43e57d7-9c37-45a0-9389-f52d40d839e2" Oct 30 18:30:05.975930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:05.975887 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" podUID="47a6c676-1f34-4d08-81e9-80051001aa80" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:30:10.727129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:10.727091 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst"] Oct 30 18:30:10.731890 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:10.731843 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" podUID="c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" containerName="istio-proxy" containerID="cri-o://e051c9b75058bc0f431667c0276eb501b4f275b14caceeb11e56e95b28a84310" gracePeriod=30 Oct 30 18:30:11.807012 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:11.806986 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" Oct 30 18:30:11.918925 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:11.918883 5232 generic.go:356] "Generic (PLEG): container finished" podID="47a6c676-1f34-4d08-81e9-80051001aa80" containerID="3aebff89f6fffffa889ccfff11a77b394bedcb71289ba904029454a339bf8385" exitCode=0 Oct 30 18:30:11.919135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:11.918948 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" Oct 30 18:30:11.919135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:11.918962 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" event={"ID":"47a6c676-1f34-4d08-81e9-80051001aa80","Type":"ContainerDied","Data":"3aebff89f6fffffa889ccfff11a77b394bedcb71289ba904029454a339bf8385"} Oct 30 18:30:11.919135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:11.919052 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6" event={"ID":"47a6c676-1f34-4d08-81e9-80051001aa80","Type":"ContainerDied","Data":"04cb40595ba7d67d12ee0c11d94440d7cec0be8335aecdccdeb252fba6f1c7fa"} Oct 30 18:30:11.919135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:11.919083 5232 scope.go:122] "RemoveContainer" containerID="3aebff89f6fffffa889ccfff11a77b394bedcb71289ba904029454a339bf8385" Oct 30 18:30:11.926775 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:11.926750 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47a6c676-1f34-4d08-81e9-80051001aa80-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a6c676-1f34-4d08-81e9-80051001aa80-plugins-config-volume\") pod \"47a6c676-1f34-4d08-81e9-80051001aa80\" (UID: \"47a6c676-1f34-4d08-81e9-80051001aa80\") " Oct 30 18:30:11.926921 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:11.926817 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47a6c676-1f34-4d08-81e9-80051001aa80-kube-api-access-nnlcc\" (UniqueName: \"kubernetes.io/projected/47a6c676-1f34-4d08-81e9-80051001aa80-kube-api-access-nnlcc\") pod \"47a6c676-1f34-4d08-81e9-80051001aa80\" (UID: \"47a6c676-1f34-4d08-81e9-80051001aa80\") " Oct 30 18:30:11.927186 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:11.927154 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47a6c676-1f34-4d08-81e9-80051001aa80-plugins-config-volume" pod "47a6c676-1f34-4d08-81e9-80051001aa80" (UID: "47a6c676-1f34-4d08-81e9-80051001aa80"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:11.929229 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:11.929203 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47a6c676-1f34-4d08-81e9-80051001aa80-kube-api-access-nnlcc" pod "47a6c676-1f34-4d08-81e9-80051001aa80" (UID: "47a6c676-1f34-4d08-81e9-80051001aa80"). InnerVolumeSpecName "kube-api-access-nnlcc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:11.930068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:11.930035 5232 scope.go:122] "RemoveContainer" containerID="3aebff89f6fffffa889ccfff11a77b394bedcb71289ba904029454a339bf8385" Oct 30 18:30:11.930523 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:11.930468 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3aebff89f6fffffa889ccfff11a77b394bedcb71289ba904029454a339bf8385\": container with ID starting with 3aebff89f6fffffa889ccfff11a77b394bedcb71289ba904029454a339bf8385 not found: ID does not exist" containerID="3aebff89f6fffffa889ccfff11a77b394bedcb71289ba904029454a339bf8385" Oct 30 18:30:11.930680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:11.930651 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3aebff89f6fffffa889ccfff11a77b394bedcb71289ba904029454a339bf8385"} err="failed to get container status \"3aebff89f6fffffa889ccfff11a77b394bedcb71289ba904029454a339bf8385\": rpc error: code = NotFound desc = could not find container \"3aebff89f6fffffa889ccfff11a77b394bedcb71289ba904029454a339bf8385\": container with ID starting with 3aebff89f6fffffa889ccfff11a77b394bedcb71289ba904029454a339bf8385 not found: ID does not exist" Oct 30 18:30:12.028561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:12.028530 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47a6c676-1f34-4d08-81e9-80051001aa80-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:12.028561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:12.028555 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nnlcc\" (UniqueName: \"kubernetes.io/projected/47a6c676-1f34-4d08-81e9-80051001aa80-kube-api-access-nnlcc\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:12.266734 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:12.266702 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6"] Oct 30 18:30:12.278752 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:12.278708 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w3-8b6cf1-5b5bd6b648-2zfj6"] Oct 30 18:30:12.802807 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:12.802766 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47a6c676-1f34-4d08-81e9-80051001aa80" path="/var/lib/kubelet/pods/47a6c676-1f34-4d08-81e9-80051001aa80/volumes" Oct 30 18:30:14.026007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:14.025979 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:14.039090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:14.039066 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:14.795603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:14.795577 5232 scope.go:122] "RemoveContainer" containerID="7df169fd24652a771524e93cfe3dc294a70d66550f3b055ac1e2911ed1249faf" Oct 30 18:30:14.954887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:14.954860 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:30:15.795741 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:15.795715 5232 scope.go:122] "RemoveContainer" containerID="1a1aee155423924283ee8bdc8e3bfcbf569b33fe81baae020f2741815035444a" Oct 30 18:30:15.796176 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:15.795849 5232 scope.go:122] "RemoveContainer" containerID="df2da92e73bb9ae809523c1cfbbcb5859268b634d33e330ad16c1fb230232a0c" Oct 30 18:30:15.961695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:15.961665 5232 generic.go:356] "Generic (PLEG): container finished" podID="d43e57d7-9c37-45a0-9389-f52d40d839e2" containerID="c65daf344b7f23bade2c3150a681d283b66e0cf589f15a670c4659bd0c2d30b9" exitCode=60 Oct 30 18:30:15.961868 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:15.961740 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" event={"ID":"d43e57d7-9c37-45a0-9389-f52d40d839e2","Type":"ContainerDied","Data":"c65daf344b7f23bade2c3150a681d283b66e0cf589f15a670c4659bd0c2d30b9"} Oct 30 18:30:15.961868 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:15.961803 5232 scope.go:122] "RemoveContainer" containerID="7df169fd24652a771524e93cfe3dc294a70d66550f3b055ac1e2911ed1249faf" Oct 30 18:30:15.962296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:15.962275 5232 scope.go:122] "RemoveContainer" containerID="c65daf344b7f23bade2c3150a681d283b66e0cf589f15a670c4659bd0c2d30b9" Oct 30 18:30:15.962663 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:15.962639 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t_odh-ai-gateway-infra(d43e57d7-9c37-45a0-9389-f52d40d839e2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" podUID="d43e57d7-9c37-45a0-9389-f52d40d839e2" Oct 30 18:30:15.964421 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:15.964399 5232 generic.go:356] "Generic (PLEG): container finished" podID="c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" containerID="e051c9b75058bc0f431667c0276eb501b4f275b14caceeb11e56e95b28a84310" exitCode=0 Oct 30 18:30:15.964524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:15.964476 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" event={"ID":"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe","Type":"ContainerDied","Data":"e051c9b75058bc0f431667c0276eb501b4f275b14caceeb11e56e95b28a84310"} Oct 30 18:30:16.082857 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.082827 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:30:16.182071 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.181979 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istiod-ca-cert\") pod \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " Oct 30 18:30:16.185237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.182071 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-workload-socket\") pod \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " Oct 30 18:30:16.185237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.182161 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-podinfo\") pod \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " Oct 30 18:30:16.185237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.182205 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-data\") pod \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " Oct 30 18:30:16.185237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.182257 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-workload-certs\") pod \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " Oct 30 18:30:16.185237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.182293 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-token\" (UniqueName: \"kubernetes.io/projected/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-token\") pod \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " Oct 30 18:30:16.185237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.182327 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-kube-api-access-2rm9t\" (UniqueName: \"kubernetes.io/projected/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-kube-api-access-2rm9t\") pod \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " Oct 30 18:30:16.185237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.182388 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-envoy\") pod \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " Oct 30 18:30:16.185237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.182403 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-workload-socket" pod "c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" (UID: "c88a6b3a-815b-4dfd-bb64-54f1c530e3fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:16.185237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.182429 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-credential-socket\") pod \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\" (UID: \"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe\") " Oct 30 18:30:16.185237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.182676 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-credential-socket" pod "c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" (UID: "c88a6b3a-815b-4dfd-bb64-54f1c530e3fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:16.185237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.183223 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.185237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.183249 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.185237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.183460 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-data" pod "c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" (UID: "c88a6b3a-815b-4dfd-bb64-54f1c530e3fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:16.190107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.186407 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istiod-ca-cert" pod "c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" (UID: "c88a6b3a-815b-4dfd-bb64-54f1c530e3fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:16.224546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.204740 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-workload-certs" pod "c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" (UID: "c88a6b3a-815b-4dfd-bb64-54f1c530e3fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:16.224546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.212068 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-podinfo" pod "c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" (UID: "c88a6b3a-815b-4dfd-bb64-54f1c530e3fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:30:16.224546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.212464 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-token" pod "c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" (UID: "c88a6b3a-815b-4dfd-bb64-54f1c530e3fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:16.224546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.213876 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-envoy" pod "c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" (UID: "c88a6b3a-815b-4dfd-bb64-54f1c530e3fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:16.253643 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.248881 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-kube-api-access-2rm9t" pod "c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" (UID: "c88a6b3a-815b-4dfd-bb64-54f1c530e3fe"). InnerVolumeSpecName "kube-api-access-2rm9t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:16.294370 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.294327 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.294370 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.294370 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.294703 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.294384 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.294703 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.294397 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.294703 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.294410 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.294703 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.294423 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rm9t\" (UniqueName: \"kubernetes.io/projected/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-kube-api-access-2rm9t\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.294703 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.294438 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:16.623631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.623600 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f"] Oct 30 18:30:16.626393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.626368 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-8b6cf1-kqq9f"] Oct 30 18:30:16.803882 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.803844 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fbe28888-b8fc-419c-a756-d417a58ebd14" path="/var/lib/kubelet/pods/fbe28888-b8fc-419c-a756-d417a58ebd14/volumes" Oct 30 18:30:16.974370 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.974333 5232 generic.go:356] "Generic (PLEG): container finished" podID="851acf02-f2a8-456f-a8a6-4cba6df59cb2" containerID="66ce38f80c38a599461ca63a1c212734ab2405ee141cf74ed92abd6616d45d17" exitCode=60 Oct 30 18:30:16.974616 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.974410 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" event={"ID":"851acf02-f2a8-456f-a8a6-4cba6df59cb2","Type":"ContainerDied","Data":"66ce38f80c38a599461ca63a1c212734ab2405ee141cf74ed92abd6616d45d17"} Oct 30 18:30:16.974616 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.974484 5232 scope.go:122] "RemoveContainer" containerID="1a1aee155423924283ee8bdc8e3bfcbf569b33fe81baae020f2741815035444a" Oct 30 18:30:16.974951 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.974933 5232 scope.go:122] "RemoveContainer" containerID="66ce38f80c38a599461ca63a1c212734ab2405ee141cf74ed92abd6616d45d17" Oct 30 18:30:16.975286 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:16.975266 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh_odh-ai-gateway-infra(851acf02-f2a8-456f-a8a6-4cba6df59cb2)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" podUID="851acf02-f2a8-456f-a8a6-4cba6df59cb2" Oct 30 18:30:16.976175 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.976157 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" Oct 30 18:30:16.976283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.976153 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst" event={"ID":"c88a6b3a-815b-4dfd-bb64-54f1c530e3fe","Type":"ContainerDied","Data":"57d057e779e6c5022b9a368799d84241ef6cf3f0582c83b40b3a241e336bb857"} Oct 30 18:30:16.978375 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.978349 5232 generic.go:356] "Generic (PLEG): container finished" podID="5e342c0b-78c7-46f5-abcf-850309d6b6cd" containerID="1508776bd405175df647e92ecaeb8fec68c55856a180ba7071aca98cafa81366" exitCode=60 Oct 30 18:30:16.978502 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.978385 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" event={"ID":"5e342c0b-78c7-46f5-abcf-850309d6b6cd","Type":"ContainerDied","Data":"1508776bd405175df647e92ecaeb8fec68c55856a180ba7071aca98cafa81366"} Oct 30 18:30:16.978818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.978795 5232 scope.go:122] "RemoveContainer" containerID="1508776bd405175df647e92ecaeb8fec68c55856a180ba7071aca98cafa81366" Oct 30 18:30:16.979092 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:16.979073 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8_odh-ai-gateway-infra(5e342c0b-78c7-46f5-abcf-850309d6b6cd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" podUID="5e342c0b-78c7-46f5-abcf-850309d6b6cd" Oct 30 18:30:16.988221 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:16.988203 5232 scope.go:122] "RemoveContainer" containerID="e051c9b75058bc0f431667c0276eb501b4f275b14caceeb11e56e95b28a84310" Oct 30 18:30:17.049536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:17.048675 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t"] Oct 30 18:30:17.095162 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:17.091834 5232 scope.go:122] "RemoveContainer" containerID="df2da92e73bb9ae809523c1cfbbcb5859268b634d33e330ad16c1fb230232a0c" Oct 30 18:30:17.125290 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:17.125260 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst"] Oct 30 18:30:17.129596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:17.128949 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-fb2fa0-openshift-default-85b9f7dfd5-f2dst"] Oct 30 18:30:17.258556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:17.258530 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" Oct 30 18:30:17.352948 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:17.352903 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d43e57d7-9c37-45a0-9389-f52d40d839e2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43e57d7-9c37-45a0-9389-f52d40d839e2-maas-api-service-ca\") pod \"d43e57d7-9c37-45a0-9389-f52d40d839e2\" (UID: \"d43e57d7-9c37-45a0-9389-f52d40d839e2\") " Oct 30 18:30:17.353153 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:17.353001 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d43e57d7-9c37-45a0-9389-f52d40d839e2-kube-api-access-pqjnj\" (UniqueName: \"kubernetes.io/projected/d43e57d7-9c37-45a0-9389-f52d40d839e2-kube-api-access-pqjnj\") pod \"d43e57d7-9c37-45a0-9389-f52d40d839e2\" (UID: \"d43e57d7-9c37-45a0-9389-f52d40d839e2\") " Oct 30 18:30:17.353264 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:17.353228 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d43e57d7-9c37-45a0-9389-f52d40d839e2-maas-api-service-ca" pod "d43e57d7-9c37-45a0-9389-f52d40d839e2" (UID: "d43e57d7-9c37-45a0-9389-f52d40d839e2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:17.353477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:17.353461 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d43e57d7-9c37-45a0-9389-f52d40d839e2-maas-api-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:17.355266 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:17.355239 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d43e57d7-9c37-45a0-9389-f52d40d839e2-kube-api-access-pqjnj" pod "d43e57d7-9c37-45a0-9389-f52d40d839e2" (UID: "d43e57d7-9c37-45a0-9389-f52d40d839e2"). InnerVolumeSpecName "kube-api-access-pqjnj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:17.455102 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:17.455018 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pqjnj\" (UniqueName: \"kubernetes.io/projected/d43e57d7-9c37-45a0-9389-f52d40d839e2-kube-api-access-pqjnj\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:17.795849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:17.795814 5232 scope.go:122] "RemoveContainer" containerID="aab66aca7ce72747bf780fd6932946661dff2c3c7bddb113a98de8c4db0b76ad" Oct 30 18:30:17.992490 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:17.992464 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" Oct 30 18:30:17.992982 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:17.992496 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t" event={"ID":"d43e57d7-9c37-45a0-9389-f52d40d839e2","Type":"ContainerDied","Data":"e3483382be43906bf8d39b9ae3ebb2f139ad4a1ab03efc0fa72d8d35d3784219"} Oct 30 18:30:17.992982 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:17.992570 5232 scope.go:122] "RemoveContainer" containerID="c65daf344b7f23bade2c3150a681d283b66e0cf589f15a670c4659bd0c2d30b9" Oct 30 18:30:18.027987 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.027961 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh"] Oct 30 18:30:18.056466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.056388 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t"] Oct 30 18:30:18.074814 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.074783 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-fde2c5-29889750-dhv9t"] Oct 30 18:30:18.082389 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.082361 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8"] Oct 30 18:30:18.203449 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.203411 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" Oct 30 18:30:18.296423 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.296392 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" Oct 30 18:30:18.366446 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.366347 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/851acf02-f2a8-456f-a8a6-4cba6df59cb2-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/851acf02-f2a8-456f-a8a6-4cba6df59cb2-maas-api-service-ca\") pod \"851acf02-f2a8-456f-a8a6-4cba6df59cb2\" (UID: \"851acf02-f2a8-456f-a8a6-4cba6df59cb2\") " Oct 30 18:30:18.366655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.366640 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/851acf02-f2a8-456f-a8a6-4cba6df59cb2-kube-api-access-d68t4\" (UniqueName: \"kubernetes.io/projected/851acf02-f2a8-456f-a8a6-4cba6df59cb2-kube-api-access-d68t4\") pod \"851acf02-f2a8-456f-a8a6-4cba6df59cb2\" (UID: \"851acf02-f2a8-456f-a8a6-4cba6df59cb2\") " Oct 30 18:30:18.366754 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.366726 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/851acf02-f2a8-456f-a8a6-4cba6df59cb2-maas-api-service-ca" pod "851acf02-f2a8-456f-a8a6-4cba6df59cb2" (UID: "851acf02-f2a8-456f-a8a6-4cba6df59cb2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:18.367116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.367095 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/851acf02-f2a8-456f-a8a6-4cba6df59cb2-maas-api-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:18.368968 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.368943 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/851acf02-f2a8-456f-a8a6-4cba6df59cb2-kube-api-access-d68t4" pod "851acf02-f2a8-456f-a8a6-4cba6df59cb2" (UID: "851acf02-f2a8-456f-a8a6-4cba6df59cb2"). InnerVolumeSpecName "kube-api-access-d68t4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:18.468440 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.468399 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e342c0b-78c7-46f5-abcf-850309d6b6cd-kube-api-access-dl7w9\" (UniqueName: \"kubernetes.io/projected/5e342c0b-78c7-46f5-abcf-850309d6b6cd-kube-api-access-dl7w9\") pod \"5e342c0b-78c7-46f5-abcf-850309d6b6cd\" (UID: \"5e342c0b-78c7-46f5-abcf-850309d6b6cd\") " Oct 30 18:30:18.468671 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.468498 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e342c0b-78c7-46f5-abcf-850309d6b6cd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e342c0b-78c7-46f5-abcf-850309d6b6cd-maas-api-service-ca\") pod \"5e342c0b-78c7-46f5-abcf-850309d6b6cd\" (UID: \"5e342c0b-78c7-46f5-abcf-850309d6b6cd\") " Oct 30 18:30:18.468852 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.468830 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e342c0b-78c7-46f5-abcf-850309d6b6cd-maas-api-service-ca" pod "5e342c0b-78c7-46f5-abcf-850309d6b6cd" (UID: "5e342c0b-78c7-46f5-abcf-850309d6b6cd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:18.468971 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.468957 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d68t4\" (UniqueName: \"kubernetes.io/projected/851acf02-f2a8-456f-a8a6-4cba6df59cb2-kube-api-access-d68t4\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:18.469023 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.468974 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/5e342c0b-78c7-46f5-abcf-850309d6b6cd-maas-api-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:18.470764 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.470743 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e342c0b-78c7-46f5-abcf-850309d6b6cd-kube-api-access-dl7w9" pod "5e342c0b-78c7-46f5-abcf-850309d6b6cd" (UID: "5e342c0b-78c7-46f5-abcf-850309d6b6cd"). InnerVolumeSpecName "kube-api-access-dl7w9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:18.570559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.570529 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dl7w9\" (UniqueName: \"kubernetes.io/projected/5e342c0b-78c7-46f5-abcf-850309d6b6cd-kube-api-access-dl7w9\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:18.803473 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.803438 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" path="/var/lib/kubelet/pods/c88a6b3a-815b-4dfd-bb64-54f1c530e3fe/volumes" Oct 30 18:30:18.803894 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:18.803879 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d43e57d7-9c37-45a0-9389-f52d40d839e2" path="/var/lib/kubelet/pods/d43e57d7-9c37-45a0-9389-f52d40d839e2/volumes" Oct 30 18:30:19.006770 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:19.006734 5232 generic.go:356] "Generic (PLEG): container finished" podID="ca70bee7-f958-4bb5-9943-f6d3f721ed21" containerID="ade00174c28696b46aa00b61089507343c451e3b4b78408e852d91bd9b0fb87e" exitCode=60 Oct 30 18:30:19.007240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:19.006797 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" event={"ID":"ca70bee7-f958-4bb5-9943-f6d3f721ed21","Type":"ContainerDied","Data":"ade00174c28696b46aa00b61089507343c451e3b4b78408e852d91bd9b0fb87e"} Oct 30 18:30:19.007240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:19.006859 5232 scope.go:122] "RemoveContainer" containerID="aab66aca7ce72747bf780fd6932946661dff2c3c7bddb113a98de8c4db0b76ad" Oct 30 18:30:19.007357 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:19.007244 5232 scope.go:122] "RemoveContainer" containerID="ade00174c28696b46aa00b61089507343c451e3b4b78408e852d91bd9b0fb87e" Oct 30 18:30:19.007820 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:19.007783 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd_odh-ai-gateway-infra(ca70bee7-f958-4bb5-9943-f6d3f721ed21)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" podUID="ca70bee7-f958-4bb5-9943-f6d3f721ed21" Oct 30 18:30:19.012953 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:19.012928 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" event={"ID":"5e342c0b-78c7-46f5-abcf-850309d6b6cd","Type":"ContainerDied","Data":"203efaa47dadb6fa3bce3a3f3264f21766f3c2116df7faaac15cd681d91f7274"} Oct 30 18:30:19.013126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:19.012941 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8" Oct 30 18:30:19.016285 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:19.016253 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" event={"ID":"851acf02-f2a8-456f-a8a6-4cba6df59cb2","Type":"ContainerDied","Data":"a1e059d9da3a8e49cdab0c17fb9d47a2bf3fcc4d5e95bc6c7f5fdae2d5fa6b6d"} Oct 30 18:30:19.016383 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:19.016346 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh" Oct 30 18:30:19.023242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:19.023223 5232 scope.go:122] "RemoveContainer" containerID="1508776bd405175df647e92ecaeb8fec68c55856a180ba7071aca98cafa81366" Oct 30 18:30:19.032581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:19.032558 5232 scope.go:122] "RemoveContainer" containerID="66ce38f80c38a599461ca63a1c212734ab2405ee141cf74ed92abd6616d45d17" Oct 30 18:30:19.096584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:19.095079 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh"] Oct 30 18:30:19.098958 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:19.098816 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-4ef6ac-29889750-58mwh"] Oct 30 18:30:19.130636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:19.130612 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8"] Oct 30 18:30:19.134910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:19.134886 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-c9a107-29889750-2n2z8"] Oct 30 18:30:20.057331 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:20.057287 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd"] Oct 30 18:30:20.244485 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:20.244435 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" Oct 30 18:30:20.399109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:20.399019 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ca70bee7-f958-4bb5-9943-f6d3f721ed21-kube-api-access-vnlf4\" (UniqueName: \"kubernetes.io/projected/ca70bee7-f958-4bb5-9943-f6d3f721ed21-kube-api-access-vnlf4\") pod \"ca70bee7-f958-4bb5-9943-f6d3f721ed21\" (UID: \"ca70bee7-f958-4bb5-9943-f6d3f721ed21\") " Oct 30 18:30:20.399109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:20.399071 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ca70bee7-f958-4bb5-9943-f6d3f721ed21-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca70bee7-f958-4bb5-9943-f6d3f721ed21-maas-api-service-ca\") pod \"ca70bee7-f958-4bb5-9943-f6d3f721ed21\" (UID: \"ca70bee7-f958-4bb5-9943-f6d3f721ed21\") " Oct 30 18:30:20.399499 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:20.399475 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ca70bee7-f958-4bb5-9943-f6d3f721ed21-maas-api-service-ca" pod "ca70bee7-f958-4bb5-9943-f6d3f721ed21" (UID: "ca70bee7-f958-4bb5-9943-f6d3f721ed21"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:20.401337 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:20.401316 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ca70bee7-f958-4bb5-9943-f6d3f721ed21-kube-api-access-vnlf4" pod "ca70bee7-f958-4bb5-9943-f6d3f721ed21" (UID: "ca70bee7-f958-4bb5-9943-f6d3f721ed21"). InnerVolumeSpecName "kube-api-access-vnlf4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:20.501319 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:20.501275 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ca70bee7-f958-4bb5-9943-f6d3f721ed21-maas-api-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:20.501319 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:20.501317 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vnlf4\" (UniqueName: \"kubernetes.io/projected/ca70bee7-f958-4bb5-9943-f6d3f721ed21-kube-api-access-vnlf4\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:20.804069 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:20.804031 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e342c0b-78c7-46f5-abcf-850309d6b6cd" path="/var/lib/kubelet/pods/5e342c0b-78c7-46f5-abcf-850309d6b6cd/volumes" Oct 30 18:30:20.804441 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:20.804424 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="851acf02-f2a8-456f-a8a6-4cba6df59cb2" path="/var/lib/kubelet/pods/851acf02-f2a8-456f-a8a6-4cba6df59cb2/volumes" Oct 30 18:30:21.042935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:21.042902 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" Oct 30 18:30:21.042935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:21.042924 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd" event={"ID":"ca70bee7-f958-4bb5-9943-f6d3f721ed21","Type":"ContainerDied","Data":"c808d32ad56d3c1236d2eab695bb648eaca20263d01261a7fba20e79a251d5a0"} Oct 30 18:30:21.043229 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:21.043001 5232 scope.go:122] "RemoveContainer" containerID="ade00174c28696b46aa00b61089507343c451e3b4b78408e852d91bd9b0fb87e" Oct 30 18:30:21.099057 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:21.099029 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd"] Oct 30 18:30:21.104231 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:21.104204 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-1b0848-29889750-2kbsd"] Oct 30 18:30:22.803113 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:22.803080 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ca70bee7-f958-4bb5-9943-f6d3f721ed21" path="/var/lib/kubelet/pods/ca70bee7-f958-4bb5-9943-f6d3f721ed21/volumes" Oct 30 18:30:25.722358 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:25.720585 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd"] Oct 30 18:30:25.722358 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:25.720814 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" podUID="f9270d40-a0b1-4c84-a28b-9f4bca0487be" containerName="istio-proxy" containerID="cri-o://2a4dc2cd673a83c04b437f0a2a42693769812dfff21b4feffd956f4ab3000743" gracePeriod=30 Oct 30 18:30:27.035613 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:27.035564 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:30:30.993132 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:30.993072 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:30:31.040179 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.040147 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-workload-socket\") pod \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " Oct 30 18:30:31.040360 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.040205 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9270d40-a0b1-4c84-a28b-9f4bca0487be-kube-api-access-rnqtx\" (UniqueName: \"kubernetes.io/projected/f9270d40-a0b1-4c84-a28b-9f4bca0487be-kube-api-access-rnqtx\") pod \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " Oct 30 18:30:31.040360 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.040332 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-envoy\") pod \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " Oct 30 18:30:31.040472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.040370 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-workload-certs\") pod \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " Oct 30 18:30:31.040472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.040409 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-data\") pod \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " Oct 30 18:30:31.040472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.040457 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-workload-socket" pod "f9270d40-a0b1-4c84-a28b-9f4bca0487be" (UID: "f9270d40-a0b1-4c84-a28b-9f4bca0487be"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:31.040686 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.040475 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istiod-ca-cert\") pod \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " Oct 30 18:30:31.040686 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.040533 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-token\" (UniqueName: \"kubernetes.io/projected/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-token\") pod \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " Oct 30 18:30:31.040792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.040712 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-workload-certs" pod "f9270d40-a0b1-4c84-a28b-9f4bca0487be" (UID: "f9270d40-a0b1-4c84-a28b-9f4bca0487be"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:31.040792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.040725 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-data" pod "f9270d40-a0b1-4c84-a28b-9f4bca0487be" (UID: "f9270d40-a0b1-4c84-a28b-9f4bca0487be"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:31.040792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.040718 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-podinfo\") pod \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " Oct 30 18:30:31.040955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.040809 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-credential-socket\") pod \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\" (UID: \"f9270d40-a0b1-4c84-a28b-9f4bca0487be\") " Oct 30 18:30:31.041160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.041096 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-credential-socket" pod "f9270d40-a0b1-4c84-a28b-9f4bca0487be" (UID: "f9270d40-a0b1-4c84-a28b-9f4bca0487be"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:31.041476 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.041268 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istiod-ca-cert" pod "f9270d40-a0b1-4c84-a28b-9f4bca0487be" (UID: "f9270d40-a0b1-4c84-a28b-9f4bca0487be"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:31.041476 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.041366 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.041476 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.041387 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.041476 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.041395 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.041476 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.041406 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.043001 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.042971 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-token" pod "f9270d40-a0b1-4c84-a28b-9f4bca0487be" (UID: "f9270d40-a0b1-4c84-a28b-9f4bca0487be"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:31.043189 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.043167 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-envoy" pod "f9270d40-a0b1-4c84-a28b-9f4bca0487be" (UID: "f9270d40-a0b1-4c84-a28b-9f4bca0487be"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:30:31.043520 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.043479 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-podinfo" pod "f9270d40-a0b1-4c84-a28b-9f4bca0487be" (UID: "f9270d40-a0b1-4c84-a28b-9f4bca0487be"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:30:31.043889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.043860 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f9270d40-a0b1-4c84-a28b-9f4bca0487be-kube-api-access-rnqtx" pod "f9270d40-a0b1-4c84-a28b-9f4bca0487be" (UID: "f9270d40-a0b1-4c84-a28b-9f4bca0487be"). InnerVolumeSpecName "kube-api-access-rnqtx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:31.116715 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.116679 5232 generic.go:356] "Generic (PLEG): container finished" podID="f9270d40-a0b1-4c84-a28b-9f4bca0487be" containerID="2a4dc2cd673a83c04b437f0a2a42693769812dfff21b4feffd956f4ab3000743" exitCode=0 Oct 30 18:30:31.116901 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.116730 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" Oct 30 18:30:31.116901 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.116755 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" event={"ID":"f9270d40-a0b1-4c84-a28b-9f4bca0487be","Type":"ContainerDied","Data":"2a4dc2cd673a83c04b437f0a2a42693769812dfff21b4feffd956f4ab3000743"} Oct 30 18:30:31.116901 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.116817 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd" event={"ID":"f9270d40-a0b1-4c84-a28b-9f4bca0487be","Type":"ContainerDied","Data":"f17c47f7938a36bf90998a1ecde43046f937a0111f8180305a1d1532901d57f8"} Oct 30 18:30:31.116901 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.116841 5232 scope.go:122] "RemoveContainer" containerID="2a4dc2cd673a83c04b437f0a2a42693769812dfff21b4feffd956f4ab3000743" Oct 30 18:30:31.129984 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.129967 5232 scope.go:122] "RemoveContainer" containerID="2a4dc2cd673a83c04b437f0a2a42693769812dfff21b4feffd956f4ab3000743" Oct 30 18:30:31.130281 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:31.130258 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a4dc2cd673a83c04b437f0a2a42693769812dfff21b4feffd956f4ab3000743\": container with ID starting with 2a4dc2cd673a83c04b437f0a2a42693769812dfff21b4feffd956f4ab3000743 not found: ID does not exist" containerID="2a4dc2cd673a83c04b437f0a2a42693769812dfff21b4feffd956f4ab3000743" Oct 30 18:30:31.130401 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.130289 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a4dc2cd673a83c04b437f0a2a42693769812dfff21b4feffd956f4ab3000743"} err="failed to get container status \"2a4dc2cd673a83c04b437f0a2a42693769812dfff21b4feffd956f4ab3000743\": rpc error: code = NotFound desc = could not find container \"2a4dc2cd673a83c04b437f0a2a42693769812dfff21b4feffd956f4ab3000743\": container with ID starting with 2a4dc2cd673a83c04b437f0a2a42693769812dfff21b4feffd956f4ab3000743 not found: ID does not exist" Oct 30 18:30:31.142662 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.142637 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.142662 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.142661 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.142833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.142671 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.142833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.142679 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f9270d40-a0b1-4c84-a28b-9f4bca0487be-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.142833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.142687 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rnqtx\" (UniqueName: \"kubernetes.io/projected/f9270d40-a0b1-4c84-a28b-9f4bca0487be-kube-api-access-rnqtx\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:31.169816 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.169786 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd"] Oct 30 18:30:31.172395 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:31.172373 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-8b6cf1-openshift-default-855578f895-wcmcd"] Oct 30 18:30:32.802898 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:32.802862 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f9270d40-a0b1-4c84-a28b-9f4bca0487be" path="/var/lib/kubelet/pods/f9270d40-a0b1-4c84-a28b-9f4bca0487be/volumes" Oct 30 18:30:34.027817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.027788 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv" Oct 30 18:30:34.103318 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.103219 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f13df76b-49f4-4942-aab5-7b0bf91ffad1-kube-api-access-n2zlv\" (UniqueName: \"kubernetes.io/projected/f13df76b-49f4-4942-aab5-7b0bf91ffad1-kube-api-access-n2zlv\") pod \"f13df76b-49f4-4942-aab5-7b0bf91ffad1\" (UID: \"f13df76b-49f4-4942-aab5-7b0bf91ffad1\") " Oct 30 18:30:34.103318 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.103309 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f13df76b-49f4-4942-aab5-7b0bf91ffad1-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13df76b-49f4-4942-aab5-7b0bf91ffad1-maas-api-service-ca\") pod \"f13df76b-49f4-4942-aab5-7b0bf91ffad1\" (UID: \"f13df76b-49f4-4942-aab5-7b0bf91ffad1\") " Oct 30 18:30:34.103808 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.103781 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f13df76b-49f4-4942-aab5-7b0bf91ffad1-maas-api-service-ca" pod "f13df76b-49f4-4942-aab5-7b0bf91ffad1" (UID: "f13df76b-49f4-4942-aab5-7b0bf91ffad1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:30:34.104047 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.104029 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f13df76b-49f4-4942-aab5-7b0bf91ffad1-maas-api-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:34.106252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.106218 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f13df76b-49f4-4942-aab5-7b0bf91ffad1-kube-api-access-n2zlv" pod "f13df76b-49f4-4942-aab5-7b0bf91ffad1" (UID: "f13df76b-49f4-4942-aab5-7b0bf91ffad1"). InnerVolumeSpecName "kube-api-access-n2zlv". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:30:34.141063 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.141034 5232 generic.go:356] "Generic (PLEG): container finished" podID="f13df76b-49f4-4942-aab5-7b0bf91ffad1" containerID="3424559a499b8a9549a854f707792468ee32242b348e359e9c8ccfc77c2844a1" exitCode=137 Oct 30 18:30:34.141242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.141085 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv" Oct 30 18:30:34.141242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.141118 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv" event={"ID":"f13df76b-49f4-4942-aab5-7b0bf91ffad1","Type":"ContainerDied","Data":"3424559a499b8a9549a854f707792468ee32242b348e359e9c8ccfc77c2844a1"} Oct 30 18:30:34.141242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.141174 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv" event={"ID":"f13df76b-49f4-4942-aab5-7b0bf91ffad1","Type":"ContainerDied","Data":"da4b173755b172c55c46e67a7ac7239a197d1be3f20d354116ced7b1ed50222b"} Oct 30 18:30:34.141242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.141196 5232 scope.go:122] "RemoveContainer" containerID="3424559a499b8a9549a854f707792468ee32242b348e359e9c8ccfc77c2844a1" Oct 30 18:30:34.151920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.151901 5232 scope.go:122] "RemoveContainer" containerID="8f47b9d82cd8bcced5dede0a116dcb7904be1ca771cda5833ba93fb274b5e10b" Oct 30 18:30:34.161339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.161320 5232 scope.go:122] "RemoveContainer" containerID="3424559a499b8a9549a854f707792468ee32242b348e359e9c8ccfc77c2844a1" Oct 30 18:30:34.161638 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:34.161618 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3424559a499b8a9549a854f707792468ee32242b348e359e9c8ccfc77c2844a1\": container with ID starting with 3424559a499b8a9549a854f707792468ee32242b348e359e9c8ccfc77c2844a1 not found: ID does not exist" containerID="3424559a499b8a9549a854f707792468ee32242b348e359e9c8ccfc77c2844a1" Oct 30 18:30:34.161681 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.161646 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3424559a499b8a9549a854f707792468ee32242b348e359e9c8ccfc77c2844a1"} err="failed to get container status \"3424559a499b8a9549a854f707792468ee32242b348e359e9c8ccfc77c2844a1\": rpc error: code = NotFound desc = could not find container \"3424559a499b8a9549a854f707792468ee32242b348e359e9c8ccfc77c2844a1\": container with ID starting with 3424559a499b8a9549a854f707792468ee32242b348e359e9c8ccfc77c2844a1 not found: ID does not exist" Oct 30 18:30:34.161681 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.161665 5232 scope.go:122] "RemoveContainer" containerID="8f47b9d82cd8bcced5dede0a116dcb7904be1ca771cda5833ba93fb274b5e10b" Oct 30 18:30:34.161944 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:34.161925 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f47b9d82cd8bcced5dede0a116dcb7904be1ca771cda5833ba93fb274b5e10b\": container with ID starting with 8f47b9d82cd8bcced5dede0a116dcb7904be1ca771cda5833ba93fb274b5e10b not found: ID does not exist" containerID="8f47b9d82cd8bcced5dede0a116dcb7904be1ca771cda5833ba93fb274b5e10b" Oct 30 18:30:34.161986 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.161951 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f47b9d82cd8bcced5dede0a116dcb7904be1ca771cda5833ba93fb274b5e10b"} err="failed to get container status \"8f47b9d82cd8bcced5dede0a116dcb7904be1ca771cda5833ba93fb274b5e10b\": rpc error: code = NotFound desc = could not find container \"8f47b9d82cd8bcced5dede0a116dcb7904be1ca771cda5833ba93fb274b5e10b\": container with ID starting with 8f47b9d82cd8bcced5dede0a116dcb7904be1ca771cda5833ba93fb274b5e10b not found: ID does not exist" Oct 30 18:30:34.193418 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.193394 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv"] Oct 30 18:30:34.198528 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.198485 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-fb2fa0-29889750-2rvmv"] Oct 30 18:30:34.205777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.205755 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2zlv\" (UniqueName: \"kubernetes.io/projected/f13df76b-49f4-4942-aab5-7b0bf91ffad1-kube-api-access-n2zlv\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:30:34.803356 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:34.803317 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f13df76b-49f4-4942-aab5-7b0bf91ffad1" path="/var/lib/kubelet/pods/f13df76b-49f4-4942-aab5-7b0bf91ffad1/volumes" Oct 30 18:30:54.693166 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:54.693131 5232 scope.go:122] "RemoveContainer" containerID="1c157233554613b7bf37167a4c343225b018f91b61688dd4bad6c683f2690c49" Oct 30 18:30:54.702959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:54.702937 5232 scope.go:122] "RemoveContainer" containerID="5276db729f7c9a48690ba54439a837dde86b28890a5b127d08ec28d76ee4db84" Oct 30 18:30:57.034105 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:30:57.034061 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:30:57.854565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:57.854532 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf"] Oct 30 18:30:57.854740 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:30:57.854706 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" podUID="b9364c01-b339-4517-b6d7-290895898b00" containerName="istio-proxy" containerID="cri-o://ddfbd1488ed0e7c097476de2d389ce8441e068549acce256e1e37729a2f558f3" gracePeriod=30 Oct 30 18:31:02.545528 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:02.545462 5232 patch_prober.go:29] interesting pod/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:31:02.546021 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:02.545543 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" podUID="b9364c01-b339-4517-b6d7-290895898b00" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:31:03.158881 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.158857 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:31:03.268502 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.268469 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-workload-socket\") pod \"b9364c01-b339-4517-b6d7-290895898b00\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " Oct 30 18:31:03.268680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.268563 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b9364c01-b339-4517-b6d7-290895898b00-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9364c01-b339-4517-b6d7-290895898b00-istiod-ca-cert\") pod \"b9364c01-b339-4517-b6d7-290895898b00\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " Oct 30 18:31:03.268680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.268607 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b9364c01-b339-4517-b6d7-290895898b00-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9364c01-b339-4517-b6d7-290895898b00-istio-podinfo\") pod \"b9364c01-b339-4517-b6d7-290895898b00\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " Oct 30 18:31:03.268680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.268649 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-istio-data\") pod \"b9364c01-b339-4517-b6d7-290895898b00\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " Oct 30 18:31:03.268680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.268675 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9364c01-b339-4517-b6d7-290895898b00-kube-api-access-4rqss\" (UniqueName: \"kubernetes.io/projected/b9364c01-b339-4517-b6d7-290895898b00-kube-api-access-4rqss\") pod \"b9364c01-b339-4517-b6d7-290895898b00\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " Oct 30 18:31:03.268907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.268710 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-workload-certs\") pod \"b9364c01-b339-4517-b6d7-290895898b00\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " Oct 30 18:31:03.268907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.268731 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-credential-socket\") pod \"b9364c01-b339-4517-b6d7-290895898b00\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " Oct 30 18:31:03.268907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.268737 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-workload-socket" pod "b9364c01-b339-4517-b6d7-290895898b00" (UID: "b9364c01-b339-4517-b6d7-290895898b00"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:03.268907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.268798 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b9364c01-b339-4517-b6d7-290895898b00-istio-token\" (UniqueName: \"kubernetes.io/projected/b9364c01-b339-4517-b6d7-290895898b00-istio-token\") pod \"b9364c01-b339-4517-b6d7-290895898b00\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " Oct 30 18:31:03.268907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.268852 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-istio-envoy\") pod \"b9364c01-b339-4517-b6d7-290895898b00\" (UID: \"b9364c01-b339-4517-b6d7-290895898b00\") " Oct 30 18:31:03.269170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.268998 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-istio-data" pod "b9364c01-b339-4517-b6d7-290895898b00" (UID: "b9364c01-b339-4517-b6d7-290895898b00"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:03.269170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.269013 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b9364c01-b339-4517-b6d7-290895898b00-istiod-ca-cert" pod "b9364c01-b339-4517-b6d7-290895898b00" (UID: "b9364c01-b339-4517-b6d7-290895898b00"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:31:03.269170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.269019 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-workload-certs" pod "b9364c01-b339-4517-b6d7-290895898b00" (UID: "b9364c01-b339-4517-b6d7-290895898b00"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:03.269170 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.269101 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-credential-socket" pod "b9364c01-b339-4517-b6d7-290895898b00" (UID: "b9364c01-b339-4517-b6d7-290895898b00"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:03.269382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.269344 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b9364c01-b339-4517-b6d7-290895898b00-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:31:03.269382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.269364 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:31:03.269382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.269375 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:31:03.269738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.269386 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:31:03.269738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.269395 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:31:03.271061 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.271032 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-istio-envoy" pod "b9364c01-b339-4517-b6d7-290895898b00" (UID: "b9364c01-b339-4517-b6d7-290895898b00"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:31:03.271197 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.271174 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9364c01-b339-4517-b6d7-290895898b00-istio-token" pod "b9364c01-b339-4517-b6d7-290895898b00" (UID: "b9364c01-b339-4517-b6d7-290895898b00"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:03.271402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.271384 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b9364c01-b339-4517-b6d7-290895898b00-istio-podinfo" pod "b9364c01-b339-4517-b6d7-290895898b00" (UID: "b9364c01-b339-4517-b6d7-290895898b00"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:31:03.271490 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.271444 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b9364c01-b339-4517-b6d7-290895898b00-kube-api-access-4rqss" pod "b9364c01-b339-4517-b6d7-290895898b00" (UID: "b9364c01-b339-4517-b6d7-290895898b00"). InnerVolumeSpecName "kube-api-access-4rqss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:03.354536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.354490 5232 generic.go:356] "Generic (PLEG): container finished" podID="b9364c01-b339-4517-b6d7-290895898b00" containerID="ddfbd1488ed0e7c097476de2d389ce8441e068549acce256e1e37729a2f558f3" exitCode=0 Oct 30 18:31:03.354707 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.354551 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" Oct 30 18:31:03.354707 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.354577 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" event={"ID":"b9364c01-b339-4517-b6d7-290895898b00","Type":"ContainerDied","Data":"ddfbd1488ed0e7c097476de2d389ce8441e068549acce256e1e37729a2f558f3"} Oct 30 18:31:03.354707 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.354634 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf" event={"ID":"b9364c01-b339-4517-b6d7-290895898b00","Type":"ContainerDied","Data":"604089073d0f795d0a93ec134750b13de21024ef3281a30796c6337247ff0378"} Oct 30 18:31:03.354707 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.354662 5232 scope.go:122] "RemoveContainer" containerID="ddfbd1488ed0e7c097476de2d389ce8441e068549acce256e1e37729a2f558f3" Oct 30 18:31:03.365880 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.365863 5232 scope.go:122] "RemoveContainer" containerID="ddfbd1488ed0e7c097476de2d389ce8441e068549acce256e1e37729a2f558f3" Oct 30 18:31:03.366153 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:31:03.366136 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ddfbd1488ed0e7c097476de2d389ce8441e068549acce256e1e37729a2f558f3\": container with ID starting with ddfbd1488ed0e7c097476de2d389ce8441e068549acce256e1e37729a2f558f3 not found: ID does not exist" containerID="ddfbd1488ed0e7c097476de2d389ce8441e068549acce256e1e37729a2f558f3" Oct 30 18:31:03.366199 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.366162 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ddfbd1488ed0e7c097476de2d389ce8441e068549acce256e1e37729a2f558f3"} err="failed to get container status \"ddfbd1488ed0e7c097476de2d389ce8441e068549acce256e1e37729a2f558f3\": rpc error: code = NotFound desc = could not find container \"ddfbd1488ed0e7c097476de2d389ce8441e068549acce256e1e37729a2f558f3\": container with ID starting with ddfbd1488ed0e7c097476de2d389ce8441e068549acce256e1e37729a2f558f3 not found: ID does not exist" Oct 30 18:31:03.370360 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.370340 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b9364c01-b339-4517-b6d7-290895898b00-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:31:03.370431 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.370363 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b9364c01-b339-4517-b6d7-290895898b00-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:31:03.370431 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.370373 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b9364c01-b339-4517-b6d7-290895898b00-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:31:03.370431 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.370381 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rqss\" (UniqueName: \"kubernetes.io/projected/b9364c01-b339-4517-b6d7-290895898b00-kube-api-access-4rqss\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:31:03.406185 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.406123 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf"] Oct 30 18:31:03.412451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:03.412429 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-4ef6ac-openshift-default-5b7984f799-7ktdf"] Oct 30 18:31:04.802047 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:04.802013 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b9364c01-b339-4517-b6d7-290895898b00" path="/var/lib/kubelet/pods/b9364c01-b339-4517-b6d7-290895898b00/volumes" Oct 30 18:31:07.031272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:07.031228 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x"] Oct 30 18:31:07.043358 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:07.043331 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c6c4a3a5-89j8x"] Oct 30 18:31:08.802376 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:08.802339 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="598f6936-6be8-480f-b11f-62fc8bc62c07" path="/var/lib/kubelet/pods/598f6936-6be8-480f-b11f-62fc8bc62c07/volumes" Oct 30 18:31:27.034463 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:31:27.034423 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:31:42.833430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.833301 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd"] Oct 30 18:31:42.837884 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.837854 5232 cpu_manager.go:414] "containerMap: removing container" podUID="47a6c676-1f34-4d08-81e9-80051001aa80" containerName="payload-processing" Oct 30 18:31:42.838116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.838094 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47a6c676-1f34-4d08-81e9-80051001aa80" containerName="payload-processing" Oct 30 18:31:42.838268 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.838256 5232 cpu_manager.go:414] "containerMap: removing container" podUID="851acf02-f2a8-456f-a8a6-4cba6df59cb2" containerName="cleanup" Oct 30 18:31:42.838358 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.838346 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="851acf02-f2a8-456f-a8a6-4cba6df59cb2" containerName="cleanup" Oct 30 18:31:42.838458 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.838448 5232 cpu_manager.go:414] "containerMap: removing container" podUID="ca70bee7-f958-4bb5-9943-f6d3f721ed21" containerName="cleanup" Oct 30 18:31:42.838558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.838547 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca70bee7-f958-4bb5-9943-f6d3f721ed21" containerName="cleanup" Oct 30 18:31:42.838650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.838641 5232 cpu_manager.go:414] "containerMap: removing container" podUID="f13df76b-49f4-4942-aab5-7b0bf91ffad1" containerName="cleanup" Oct 30 18:31:42.838725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.838715 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f13df76b-49f4-4942-aab5-7b0bf91ffad1" containerName="cleanup" Oct 30 18:31:42.838819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.838809 5232 cpu_manager.go:414] "containerMap: removing container" podUID="c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" containerName="istio-proxy" Oct 30 18:31:42.838908 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.838896 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" containerName="istio-proxy" Oct 30 18:31:42.839005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.838996 5232 cpu_manager.go:414] "containerMap: removing container" podUID="b9364c01-b339-4517-b6d7-290895898b00" containerName="istio-proxy" Oct 30 18:31:42.839080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.839070 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b9364c01-b339-4517-b6d7-290895898b00" containerName="istio-proxy" Oct 30 18:31:42.839166 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.839157 5232 cpu_manager.go:414] "containerMap: removing container" podUID="f9270d40-a0b1-4c84-a28b-9f4bca0487be" containerName="istio-proxy" Oct 30 18:31:42.839240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.839228 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f9270d40-a0b1-4c84-a28b-9f4bca0487be" containerName="istio-proxy" Oct 30 18:31:42.839324 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.839315 5232 cpu_manager.go:414] "containerMap: removing container" podUID="5e342c0b-78c7-46f5-abcf-850309d6b6cd" containerName="cleanup" Oct 30 18:31:42.839646 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.839626 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e342c0b-78c7-46f5-abcf-850309d6b6cd" containerName="cleanup" Oct 30 18:31:42.841391 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.839773 5232 cpu_manager.go:414] "containerMap: removing container" podUID="ca70bee7-f958-4bb5-9943-f6d3f721ed21" containerName="cleanup" Oct 30 18:31:42.841569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.841551 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca70bee7-f958-4bb5-9943-f6d3f721ed21" containerName="cleanup" Oct 30 18:31:42.841756 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.841741 5232 cpu_manager.go:414] "containerMap: removing container" podUID="d43e57d7-9c37-45a0-9389-f52d40d839e2" containerName="cleanup" Oct 30 18:31:42.843063 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.843046 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43e57d7-9c37-45a0-9389-f52d40d839e2" containerName="cleanup" Oct 30 18:31:42.843165 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.843153 5232 cpu_manager.go:414] "containerMap: removing container" podUID="391ee642-1ab6-47cc-b103-f381ea9fd5bd" containerName="revoke-keys" Oct 30 18:31:42.843255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.843244 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="391ee642-1ab6-47cc-b103-f381ea9fd5bd" containerName="revoke-keys" Oct 30 18:31:42.843336 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.843327 5232 cpu_manager.go:414] "containerMap: removing container" podUID="ca70bee7-f958-4bb5-9943-f6d3f721ed21" containerName="cleanup" Oct 30 18:31:42.843415 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.843406 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ca70bee7-f958-4bb5-9943-f6d3f721ed21" containerName="cleanup" Oct 30 18:31:42.843495 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.843486 5232 cpu_manager.go:414] "containerMap: removing container" podUID="d43e57d7-9c37-45a0-9389-f52d40d839e2" containerName="cleanup" Oct 30 18:31:42.843596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.843584 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43e57d7-9c37-45a0-9389-f52d40d839e2" containerName="cleanup" Oct 30 18:31:42.843692 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.843683 5232 cpu_manager.go:414] "containerMap: removing container" podUID="5e342c0b-78c7-46f5-abcf-850309d6b6cd" containerName="cleanup" Oct 30 18:31:42.843763 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.843754 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e342c0b-78c7-46f5-abcf-850309d6b6cd" containerName="cleanup" Oct 30 18:31:42.843853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.843844 5232 cpu_manager.go:414] "containerMap: removing container" podUID="d43e57d7-9c37-45a0-9389-f52d40d839e2" containerName="cleanup" Oct 30 18:31:42.843929 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.843920 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d43e57d7-9c37-45a0-9389-f52d40d839e2" containerName="cleanup" Oct 30 18:31:42.844015 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844007 5232 cpu_manager.go:414] "containerMap: removing container" podUID="851acf02-f2a8-456f-a8a6-4cba6df59cb2" containerName="cleanup" Oct 30 18:31:42.844089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844079 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="851acf02-f2a8-456f-a8a6-4cba6df59cb2" containerName="cleanup" Oct 30 18:31:42.844328 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844305 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="851acf02-f2a8-456f-a8a6-4cba6df59cb2" containerName="cleanup" Oct 30 18:31:42.844328 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844329 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="f9270d40-a0b1-4c84-a28b-9f4bca0487be" containerName="istio-proxy" Oct 30 18:31:42.844491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844340 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e342c0b-78c7-46f5-abcf-850309d6b6cd" containerName="cleanup" Oct 30 18:31:42.844491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844349 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca70bee7-f958-4bb5-9943-f6d3f721ed21" containerName="cleanup" Oct 30 18:31:42.844491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844360 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43e57d7-9c37-45a0-9389-f52d40d839e2" containerName="cleanup" Oct 30 18:31:42.844491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844367 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="391ee642-1ab6-47cc-b103-f381ea9fd5bd" containerName="revoke-keys" Oct 30 18:31:42.844491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844375 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca70bee7-f958-4bb5-9943-f6d3f721ed21" containerName="cleanup" Oct 30 18:31:42.844491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844382 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43e57d7-9c37-45a0-9389-f52d40d839e2" containerName="cleanup" Oct 30 18:31:42.844491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844393 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e342c0b-78c7-46f5-abcf-850309d6b6cd" containerName="cleanup" Oct 30 18:31:42.844491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844402 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e342c0b-78c7-46f5-abcf-850309d6b6cd" containerName="cleanup" Oct 30 18:31:42.844491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844409 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="d43e57d7-9c37-45a0-9389-f52d40d839e2" containerName="cleanup" Oct 30 18:31:42.844491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844416 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="f13df76b-49f4-4942-aab5-7b0bf91ffad1" containerName="cleanup" Oct 30 18:31:42.844491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844423 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="851acf02-f2a8-456f-a8a6-4cba6df59cb2" containerName="cleanup" Oct 30 18:31:42.844491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844434 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="47a6c676-1f34-4d08-81e9-80051001aa80" containerName="payload-processing" Oct 30 18:31:42.844491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844441 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="851acf02-f2a8-456f-a8a6-4cba6df59cb2" containerName="cleanup" Oct 30 18:31:42.844491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844450 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="f13df76b-49f4-4942-aab5-7b0bf91ffad1" containerName="cleanup" Oct 30 18:31:42.844491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844461 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="c88a6b3a-815b-4dfd-bb64-54f1c530e3fe" containerName="istio-proxy" Oct 30 18:31:42.844491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.844468 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="b9364c01-b339-4517-b6d7-290895898b00" containerName="istio-proxy" Oct 30 18:31:42.848445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.848213 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd"] Oct 30 18:31:42.848445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.848314 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd" Oct 30 18:31:42.875823 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.875795 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:31:42.875984 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.875964 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:31:42.973451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.973419 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f12b025-rp8gd\" (UID: \"31ff171d-ddfa-4212-b9eb-8c47d790a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd" Oct 30 18:31:42.973647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.973485 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f12b025-rp8gd\" (UID: \"31ff171d-ddfa-4212-b9eb-8c47d790a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd" Oct 30 18:31:42.973647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:42.973636 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nc4t\" (UniqueName: \"kubernetes.io/projected/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-kube-api-access-9nc4t\") pod \"maas-api-revoke-sub-8f12b025-rp8gd\" (UID: \"31ff171d-ddfa-4212-b9eb-8c47d790a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd" Oct 30 18:31:43.075143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:43.075109 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f12b025-rp8gd\" (UID: \"31ff171d-ddfa-4212-b9eb-8c47d790a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd" Oct 30 18:31:43.075347 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:43.075239 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nc4t\" (UniqueName: \"kubernetes.io/projected/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-kube-api-access-9nc4t\") pod \"maas-api-revoke-sub-8f12b025-rp8gd\" (UID: \"31ff171d-ddfa-4212-b9eb-8c47d790a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd" Oct 30 18:31:43.075347 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:43.075293 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f12b025-rp8gd\" (UID: \"31ff171d-ddfa-4212-b9eb-8c47d790a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd" Oct 30 18:31:43.075960 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:43.075936 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8f12b025-rp8gd\" (UID: \"31ff171d-ddfa-4212-b9eb-8c47d790a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd" Oct 30 18:31:43.077849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:43.077823 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8f12b025-rp8gd\" (UID: \"31ff171d-ddfa-4212-b9eb-8c47d790a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd" Oct 30 18:31:43.095132 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:43.095077 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nc4t\" (UniqueName: \"kubernetes.io/projected/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-kube-api-access-9nc4t\") pod \"maas-api-revoke-sub-8f12b025-rp8gd\" (UID: \"31ff171d-ddfa-4212-b9eb-8c47d790a3ec\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd" Oct 30 18:31:43.181077 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:43.181053 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd" Oct 30 18:31:43.476033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:43.476006 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd"] Oct 30 18:31:43.476693 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:31:43.476657 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31ff171d_ddfa_4212_b9eb_8c47d790a3ec.slice/crio-a099dfad22f4c046b4e45e49d64e7c0d80769fe4dd88ad54079a3a3906786af0 WatchSource:0}: Error finding container a099dfad22f4c046b4e45e49d64e7c0d80769fe4dd88ad54079a3a3906786af0: Status 404 returned error can't find the container with id a099dfad22f4c046b4e45e49d64e7c0d80769fe4dd88ad54079a3a3906786af0 Oct 30 18:31:43.629753 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:43.629709 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd" event={"ID":"31ff171d-ddfa-4212-b9eb-8c47d790a3ec","Type":"ContainerStarted","Data":"a099dfad22f4c046b4e45e49d64e7c0d80769fe4dd88ad54079a3a3906786af0"} Oct 30 18:31:44.639397 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:44.639311 5232 generic.go:356] "Generic (PLEG): container finished" podID="31ff171d-ddfa-4212-b9eb-8c47d790a3ec" containerID="7e79a8b167ba9698c0776c0cb58696990c3cd2d13192427f88a981e6e4a8dbcb" exitCode=0 Oct 30 18:31:44.639828 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:44.639400 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd" event={"ID":"31ff171d-ddfa-4212-b9eb-8c47d790a3ec","Type":"ContainerDied","Data":"7e79a8b167ba9698c0776c0cb58696990c3cd2d13192427f88a981e6e4a8dbcb"} Oct 30 18:31:45.790619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:45.790595 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd" Oct 30 18:31:45.907182 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:45.907101 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-maas-api-openshift-ca\") pod \"31ff171d-ddfa-4212-b9eb-8c47d790a3ec\" (UID: \"31ff171d-ddfa-4212-b9eb-8c47d790a3ec\") " Oct 30 18:31:45.907355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:45.907210 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-kube-api-access-9nc4t\" (UniqueName: \"kubernetes.io/projected/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-kube-api-access-9nc4t\") pod \"31ff171d-ddfa-4212-b9eb-8c47d790a3ec\" (UID: \"31ff171d-ddfa-4212-b9eb-8c47d790a3ec\") " Oct 30 18:31:45.907423 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:45.907368 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-maas-api-xks-ca\") pod \"31ff171d-ddfa-4212-b9eb-8c47d790a3ec\" (UID: \"31ff171d-ddfa-4212-b9eb-8c47d790a3ec\") " Oct 30 18:31:45.907764 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:45.907731 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-maas-api-openshift-ca" pod "31ff171d-ddfa-4212-b9eb-8c47d790a3ec" (UID: "31ff171d-ddfa-4212-b9eb-8c47d790a3ec"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:31:45.909577 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:45.909547 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-maas-api-xks-ca" pod "31ff171d-ddfa-4212-b9eb-8c47d790a3ec" (UID: "31ff171d-ddfa-4212-b9eb-8c47d790a3ec"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:31:45.909577 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:45.909559 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-kube-api-access-9nc4t" pod "31ff171d-ddfa-4212-b9eb-8c47d790a3ec" (UID: "31ff171d-ddfa-4212-b9eb-8c47d790a3ec"). InnerVolumeSpecName "kube-api-access-9nc4t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:46.012536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:46.012345 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nc4t\" (UniqueName: \"kubernetes.io/projected/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-kube-api-access-9nc4t\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:31:46.012536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:46.012389 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:31:46.012536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:46.012404 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/31ff171d-ddfa-4212-b9eb-8c47d790a3ec-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:31:46.654409 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:46.654381 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd" Oct 30 18:31:46.654660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:46.654420 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd" event={"ID":"31ff171d-ddfa-4212-b9eb-8c47d790a3ec","Type":"ContainerDied","Data":"a099dfad22f4c046b4e45e49d64e7c0d80769fe4dd88ad54079a3a3906786af0"} Oct 30 18:31:46.654660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:46.654478 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a099dfad22f4c046b4e45e49d64e7c0d80769fe4dd88ad54079a3a3906786af0" Oct 30 18:31:50.346350 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.346304 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp"] Oct 30 18:31:50.346865 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.346827 5232 cpu_manager.go:414] "containerMap: removing container" podUID="851acf02-f2a8-456f-a8a6-4cba6df59cb2" containerName="cleanup" Oct 30 18:31:50.346865 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.346839 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="851acf02-f2a8-456f-a8a6-4cba6df59cb2" containerName="cleanup" Oct 30 18:31:50.346865 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.346853 5232 cpu_manager.go:414] "containerMap: removing container" podUID="f13df76b-49f4-4942-aab5-7b0bf91ffad1" containerName="cleanup" Oct 30 18:31:50.346865 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.346858 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f13df76b-49f4-4942-aab5-7b0bf91ffad1" containerName="cleanup" Oct 30 18:31:50.347003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.346872 5232 cpu_manager.go:414] "containerMap: removing container" podUID="31ff171d-ddfa-4212-b9eb-8c47d790a3ec" containerName="revoke-keys" Oct 30 18:31:50.347003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.346877 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="31ff171d-ddfa-4212-b9eb-8c47d790a3ec" containerName="revoke-keys" Oct 30 18:31:50.347003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.346900 5232 cpu_manager.go:414] "containerMap: removing container" podUID="5e342c0b-78c7-46f5-abcf-850309d6b6cd" containerName="cleanup" Oct 30 18:31:50.347003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.346905 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e342c0b-78c7-46f5-abcf-850309d6b6cd" containerName="cleanup" Oct 30 18:31:50.347003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.346983 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="ca70bee7-f958-4bb5-9943-f6d3f721ed21" containerName="cleanup" Oct 30 18:31:50.347003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.346998 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="31ff171d-ddfa-4212-b9eb-8c47d790a3ec" containerName="revoke-keys" Oct 30 18:31:50.350748 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.350732 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp" Oct 30 18:31:50.362024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.362002 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07405f12-8a6c-4a2d-af64-70650200a1cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b574ab3-x4kkp\" (UID: \"07405f12-8a6c-4a2d-af64-70650200a1cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp" Oct 30 18:31:50.362125 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.362037 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dmgb\" (UniqueName: \"kubernetes.io/projected/07405f12-8a6c-4a2d-af64-70650200a1cf-kube-api-access-2dmgb\") pod \"maas-api-revoke-sub-8b574ab3-x4kkp\" (UID: \"07405f12-8a6c-4a2d-af64-70650200a1cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp" Oct 30 18:31:50.362220 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.362168 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07405f12-8a6c-4a2d-af64-70650200a1cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b574ab3-x4kkp\" (UID: \"07405f12-8a6c-4a2d-af64-70650200a1cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp" Oct 30 18:31:50.369597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.369570 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp"] Oct 30 18:31:50.377957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.377937 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:31:50.378097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.377989 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:31:50.463084 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.463056 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07405f12-8a6c-4a2d-af64-70650200a1cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b574ab3-x4kkp\" (UID: \"07405f12-8a6c-4a2d-af64-70650200a1cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp" Oct 30 18:31:50.463227 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.463096 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dmgb\" (UniqueName: \"kubernetes.io/projected/07405f12-8a6c-4a2d-af64-70650200a1cf-kube-api-access-2dmgb\") pod \"maas-api-revoke-sub-8b574ab3-x4kkp\" (UID: \"07405f12-8a6c-4a2d-af64-70650200a1cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp" Oct 30 18:31:50.463227 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.463204 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07405f12-8a6c-4a2d-af64-70650200a1cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b574ab3-x4kkp\" (UID: \"07405f12-8a6c-4a2d-af64-70650200a1cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp" Oct 30 18:31:50.463873 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.463854 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07405f12-8a6c-4a2d-af64-70650200a1cf-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-8b574ab3-x4kkp\" (UID: \"07405f12-8a6c-4a2d-af64-70650200a1cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp" Oct 30 18:31:50.465572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.465553 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07405f12-8a6c-4a2d-af64-70650200a1cf-maas-api-xks-ca\") pod \"maas-api-revoke-sub-8b574ab3-x4kkp\" (UID: \"07405f12-8a6c-4a2d-af64-70650200a1cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp" Oct 30 18:31:50.482666 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.482641 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dmgb\" (UniqueName: \"kubernetes.io/projected/07405f12-8a6c-4a2d-af64-70650200a1cf-kube-api-access-2dmgb\") pod \"maas-api-revoke-sub-8b574ab3-x4kkp\" (UID: \"07405f12-8a6c-4a2d-af64-70650200a1cf\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp" Oct 30 18:31:50.664096 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.663981 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp" Oct 30 18:31:50.962052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:50.962016 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp"] Oct 30 18:31:50.962609 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:31:50.962579 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07405f12_8a6c_4a2d_af64_70650200a1cf.slice/crio-ffe67a156041f011e7ab609c3ae7545e1ad5f944cf0bf102d4b56706ce6b468c WatchSource:0}: Error finding container ffe67a156041f011e7ab609c3ae7545e1ad5f944cf0bf102d4b56706ce6b468c: Status 404 returned error can't find the container with id ffe67a156041f011e7ab609c3ae7545e1ad5f944cf0bf102d4b56706ce6b468c Oct 30 18:31:51.690498 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:51.690452 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp" event={"ID":"07405f12-8a6c-4a2d-af64-70650200a1cf","Type":"ContainerStarted","Data":"6bebe22962bdfeb4f3f3d0cf99fb4e5d3c37e56da15c9241cd31472c59a7613a"} Oct 30 18:31:51.690948 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:51.690548 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp" event={"ID":"07405f12-8a6c-4a2d-af64-70650200a1cf","Type":"ContainerStarted","Data":"ffe67a156041f011e7ab609c3ae7545e1ad5f944cf0bf102d4b56706ce6b468c"} Oct 30 18:31:51.729980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:51.729926 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp" podStartSLOduration=1.729913179 podStartE2EDuration="1.729913179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:31:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:31:51.728772877 +0000 UTC m=+1811.045193197" watchObservedRunningTime="2026-10-30 18:31:51.729913179 +0000 UTC m=+1811.046333499" Oct 30 18:31:52.698821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:52.698780 5232 generic.go:356] "Generic (PLEG): container finished" podID="07405f12-8a6c-4a2d-af64-70650200a1cf" containerID="6bebe22962bdfeb4f3f3d0cf99fb4e5d3c37e56da15c9241cd31472c59a7613a" exitCode=0 Oct 30 18:31:52.699260 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:52.698873 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp" event={"ID":"07405f12-8a6c-4a2d-af64-70650200a1cf","Type":"ContainerDied","Data":"6bebe22962bdfeb4f3f3d0cf99fb4e5d3c37e56da15c9241cd31472c59a7613a"} Oct 30 18:31:53.775653 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:53.775607 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5"] Oct 30 18:31:53.909822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:53.909792 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp" Oct 30 18:31:54.104651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.104618 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/07405f12-8a6c-4a2d-af64-70650200a1cf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07405f12-8a6c-4a2d-af64-70650200a1cf-maas-api-xks-ca\") pod \"07405f12-8a6c-4a2d-af64-70650200a1cf\" (UID: \"07405f12-8a6c-4a2d-af64-70650200a1cf\") " Oct 30 18:31:54.104847 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.104755 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07405f12-8a6c-4a2d-af64-70650200a1cf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07405f12-8a6c-4a2d-af64-70650200a1cf-maas-api-openshift-ca\") pod \"07405f12-8a6c-4a2d-af64-70650200a1cf\" (UID: \"07405f12-8a6c-4a2d-af64-70650200a1cf\") " Oct 30 18:31:54.104931 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.104843 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07405f12-8a6c-4a2d-af64-70650200a1cf-kube-api-access-2dmgb\" (UniqueName: \"kubernetes.io/projected/07405f12-8a6c-4a2d-af64-70650200a1cf-kube-api-access-2dmgb\") pod \"07405f12-8a6c-4a2d-af64-70650200a1cf\" (UID: \"07405f12-8a6c-4a2d-af64-70650200a1cf\") " Oct 30 18:31:54.105157 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.105119 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07405f12-8a6c-4a2d-af64-70650200a1cf-maas-api-openshift-ca" pod "07405f12-8a6c-4a2d-af64-70650200a1cf" (UID: "07405f12-8a6c-4a2d-af64-70650200a1cf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:31:54.105571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.105551 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/07405f12-8a6c-4a2d-af64-70650200a1cf-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:31:54.107073 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.107049 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/07405f12-8a6c-4a2d-af64-70650200a1cf-maas-api-xks-ca" pod "07405f12-8a6c-4a2d-af64-70650200a1cf" (UID: "07405f12-8a6c-4a2d-af64-70650200a1cf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:31:54.107189 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.107074 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07405f12-8a6c-4a2d-af64-70650200a1cf-kube-api-access-2dmgb" pod "07405f12-8a6c-4a2d-af64-70650200a1cf" (UID: "07405f12-8a6c-4a2d-af64-70650200a1cf"). InnerVolumeSpecName "kube-api-access-2dmgb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:54.207017 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.206985 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dmgb\" (UniqueName: \"kubernetes.io/projected/07405f12-8a6c-4a2d-af64-70650200a1cf-kube-api-access-2dmgb\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:31:54.207017 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.207018 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/07405f12-8a6c-4a2d-af64-70650200a1cf-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:31:54.329985 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.329945 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82"] Oct 30 18:31:54.330616 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.330597 5232 cpu_manager.go:414] "containerMap: removing container" podUID="07405f12-8a6c-4a2d-af64-70650200a1cf" containerName="revoke-keys" Oct 30 18:31:54.330616 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.330615 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07405f12-8a6c-4a2d-af64-70650200a1cf" containerName="revoke-keys" Oct 30 18:31:54.330811 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.330736 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="07405f12-8a6c-4a2d-af64-70650200a1cf" containerName="revoke-keys" Oct 30 18:31:54.334183 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.334165 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82" Oct 30 18:31:54.352140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.352117 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82"] Oct 30 18:31:54.411016 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.410941 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f06bdf8-de96-4dce-8971-810550144958-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad1ec265-5zp82\" (UID: \"5f06bdf8-de96-4dce-8971-810550144958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82" Oct 30 18:31:54.411016 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.411005 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f06bdf8-de96-4dce-8971-810550144958-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad1ec265-5zp82\" (UID: \"5f06bdf8-de96-4dce-8971-810550144958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82" Oct 30 18:31:54.411179 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.411164 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nblkq\" (UniqueName: \"kubernetes.io/projected/5f06bdf8-de96-4dce-8971-810550144958-kube-api-access-nblkq\") pod \"maas-api-revoke-sub-ad1ec265-5zp82\" (UID: \"5f06bdf8-de96-4dce-8971-810550144958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82" Oct 30 18:31:54.512455 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.512412 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f06bdf8-de96-4dce-8971-810550144958-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad1ec265-5zp82\" (UID: \"5f06bdf8-de96-4dce-8971-810550144958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82" Oct 30 18:31:54.512650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.512563 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nblkq\" (UniqueName: \"kubernetes.io/projected/5f06bdf8-de96-4dce-8971-810550144958-kube-api-access-nblkq\") pod \"maas-api-revoke-sub-ad1ec265-5zp82\" (UID: \"5f06bdf8-de96-4dce-8971-810550144958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82" Oct 30 18:31:54.512650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.512633 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f06bdf8-de96-4dce-8971-810550144958-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad1ec265-5zp82\" (UID: \"5f06bdf8-de96-4dce-8971-810550144958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82" Oct 30 18:31:54.513111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.513086 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f06bdf8-de96-4dce-8971-810550144958-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ad1ec265-5zp82\" (UID: \"5f06bdf8-de96-4dce-8971-810550144958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82" Oct 30 18:31:54.515153 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.515129 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f06bdf8-de96-4dce-8971-810550144958-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ad1ec265-5zp82\" (UID: \"5f06bdf8-de96-4dce-8971-810550144958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82" Oct 30 18:31:54.532318 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.532297 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nblkq\" (UniqueName: \"kubernetes.io/projected/5f06bdf8-de96-4dce-8971-810550144958-kube-api-access-nblkq\") pod \"maas-api-revoke-sub-ad1ec265-5zp82\" (UID: \"5f06bdf8-de96-4dce-8971-810550144958\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82" Oct 30 18:31:54.647589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.647561 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82" Oct 30 18:31:54.716937 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.716904 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp" event={"ID":"07405f12-8a6c-4a2d-af64-70650200a1cf","Type":"ContainerDied","Data":"ffe67a156041f011e7ab609c3ae7545e1ad5f944cf0bf102d4b56706ce6b468c"} Oct 30 18:31:54.717087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.716964 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ffe67a156041f011e7ab609c3ae7545e1ad5f944cf0bf102d4b56706ce6b468c" Oct 30 18:31:54.717087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.717036 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp" Oct 30 18:31:54.796208 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.796172 5232 scope.go:122] "RemoveContainer" containerID="790a2963ad1401ef42558e95474b87268990b93e611c14c11320ab2845b39370" Oct 30 18:31:54.807027 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.807008 5232 scope.go:122] "RemoveContainer" containerID="5900e78954006843ceacdb34862b0d57d856c9772b93fe0a3414497a5db57085" Oct 30 18:31:54.819086 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.819063 5232 scope.go:122] "RemoveContainer" containerID="38ec60b5bfdba81bcaf0808cae8919650875c1d74c3ce3863a8218858f60d018" Oct 30 18:31:54.828111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.828093 5232 scope.go:122] "RemoveContainer" containerID="67120584dae0790243e7e6a94063704bf18b6414b398e4c9a342526db1575c9b" Oct 30 18:31:54.836455 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.836436 5232 scope.go:122] "RemoveContainer" containerID="29faeca69bacad3df2b82e9587c9805604d3b8d39ce4e1689d854017f6c3a01a" Oct 30 18:31:54.943463 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:54.943437 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82"] Oct 30 18:31:54.943858 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:31:54.943835 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f06bdf8_de96_4dce_8971_810550144958.slice/crio-d4539402899bfdf445b185a615fce923755f189f5b7f2e96a527a5edcb37c562 WatchSource:0}: Error finding container d4539402899bfdf445b185a615fce923755f189f5b7f2e96a527a5edcb37c562: Status 404 returned error can't find the container with id d4539402899bfdf445b185a615fce923755f189f5b7f2e96a527a5edcb37c562 Oct 30 18:31:55.726600 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:55.726565 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82" event={"ID":"5f06bdf8-de96-4dce-8971-810550144958","Type":"ContainerStarted","Data":"09cd5f64eb7617cf8ec190ced5ed13edccc686451ca466c8098c3f97a8b038a6"} Oct 30 18:31:55.726770 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:55.726617 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82" event={"ID":"5f06bdf8-de96-4dce-8971-810550144958","Type":"ContainerStarted","Data":"d4539402899bfdf445b185a615fce923755f189f5b7f2e96a527a5edcb37c562"} Oct 30 18:31:56.735636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:56.735593 5232 generic.go:356] "Generic (PLEG): container finished" podID="5f06bdf8-de96-4dce-8971-810550144958" containerID="09cd5f64eb7617cf8ec190ced5ed13edccc686451ca466c8098c3f97a8b038a6" exitCode=0 Oct 30 18:31:56.736075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:56.735675 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82" event={"ID":"5f06bdf8-de96-4dce-8971-810550144958","Type":"ContainerDied","Data":"09cd5f64eb7617cf8ec190ced5ed13edccc686451ca466c8098c3f97a8b038a6"} Oct 30 18:31:57.034344 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:31:57.034263 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:31:57.969391 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:57.969362 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82" Oct 30 18:31:58.050387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:58.050317 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f06bdf8-de96-4dce-8971-810550144958-kube-api-access-nblkq\" (UniqueName: \"kubernetes.io/projected/5f06bdf8-de96-4dce-8971-810550144958-kube-api-access-nblkq\") pod \"5f06bdf8-de96-4dce-8971-810550144958\" (UID: \"5f06bdf8-de96-4dce-8971-810550144958\") " Oct 30 18:31:58.050568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:58.050401 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5f06bdf8-de96-4dce-8971-810550144958-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f06bdf8-de96-4dce-8971-810550144958-maas-api-xks-ca\") pod \"5f06bdf8-de96-4dce-8971-810550144958\" (UID: \"5f06bdf8-de96-4dce-8971-810550144958\") " Oct 30 18:31:58.050568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:58.050453 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f06bdf8-de96-4dce-8971-810550144958-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f06bdf8-de96-4dce-8971-810550144958-maas-api-openshift-ca\") pod \"5f06bdf8-de96-4dce-8971-810550144958\" (UID: \"5f06bdf8-de96-4dce-8971-810550144958\") " Oct 30 18:31:58.050884 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:58.050853 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f06bdf8-de96-4dce-8971-810550144958-maas-api-openshift-ca" pod "5f06bdf8-de96-4dce-8971-810550144958" (UID: "5f06bdf8-de96-4dce-8971-810550144958"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:31:58.052792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:58.052767 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f06bdf8-de96-4dce-8971-810550144958-kube-api-access-nblkq" pod "5f06bdf8-de96-4dce-8971-810550144958" (UID: "5f06bdf8-de96-4dce-8971-810550144958"). InnerVolumeSpecName "kube-api-access-nblkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:31:58.053010 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:58.052994 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5f06bdf8-de96-4dce-8971-810550144958-maas-api-xks-ca" pod "5f06bdf8-de96-4dce-8971-810550144958" (UID: "5f06bdf8-de96-4dce-8971-810550144958"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:31:58.152501 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:58.152463 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5f06bdf8-de96-4dce-8971-810550144958-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:31:58.152669 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:58.152566 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5f06bdf8-de96-4dce-8971-810550144958-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:31:58.152669 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:58.152587 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nblkq\" (UniqueName: \"kubernetes.io/projected/5f06bdf8-de96-4dce-8971-810550144958-kube-api-access-nblkq\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:31:58.752831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:58.752799 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82" Oct 30 18:31:58.752831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:58.752822 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82" event={"ID":"5f06bdf8-de96-4dce-8971-810550144958","Type":"ContainerDied","Data":"d4539402899bfdf445b185a615fce923755f189f5b7f2e96a527a5edcb37c562"} Oct 30 18:31:58.753058 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:31:58.752867 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d4539402899bfdf445b185a615fce923755f189f5b7f2e96a527a5edcb37c562" Oct 30 18:32:06.898116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:06.898084 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv"] Oct 30 18:32:07.491544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:07.491496 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb"] Oct 30 18:32:07.492189 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:07.492169 5232 cpu_manager.go:414] "containerMap: removing container" podUID="5f06bdf8-de96-4dce-8971-810550144958" containerName="revoke-keys" Oct 30 18:32:07.492250 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:07.492192 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f06bdf8-de96-4dce-8971-810550144958" containerName="revoke-keys" Oct 30 18:32:07.492306 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:07.492297 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f06bdf8-de96-4dce-8971-810550144958" containerName="revoke-keys" Oct 30 18:32:07.495678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:07.495662 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb" Oct 30 18:32:07.511448 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:07.511428 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:32:07.511707 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:07.511690 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:32:07.527448 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:07.527417 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb"] Oct 30 18:32:07.562291 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:07.562263 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62472024-eb2f-418e-a074-acb6bf6b46c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42b56365-tf5kb\" (UID: \"62472024-eb2f-418e-a074-acb6bf6b46c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb" Oct 30 18:32:07.562461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:07.562303 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zjbkw\" (UniqueName: \"kubernetes.io/projected/62472024-eb2f-418e-a074-acb6bf6b46c6-kube-api-access-zjbkw\") pod \"maas-api-revoke-sub-42b56365-tf5kb\" (UID: \"62472024-eb2f-418e-a074-acb6bf6b46c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb" Oct 30 18:32:07.562461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:07.562439 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62472024-eb2f-418e-a074-acb6bf6b46c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42b56365-tf5kb\" (UID: \"62472024-eb2f-418e-a074-acb6bf6b46c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb" Oct 30 18:32:07.664074 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:07.664044 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62472024-eb2f-418e-a074-acb6bf6b46c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42b56365-tf5kb\" (UID: \"62472024-eb2f-418e-a074-acb6bf6b46c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb" Oct 30 18:32:07.664262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:07.664166 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62472024-eb2f-418e-a074-acb6bf6b46c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42b56365-tf5kb\" (UID: \"62472024-eb2f-418e-a074-acb6bf6b46c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb" Oct 30 18:32:07.664262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:07.664190 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zjbkw\" (UniqueName: \"kubernetes.io/projected/62472024-eb2f-418e-a074-acb6bf6b46c6-kube-api-access-zjbkw\") pod \"maas-api-revoke-sub-42b56365-tf5kb\" (UID: \"62472024-eb2f-418e-a074-acb6bf6b46c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb" Oct 30 18:32:07.664869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:07.664840 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62472024-eb2f-418e-a074-acb6bf6b46c6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-42b56365-tf5kb\" (UID: \"62472024-eb2f-418e-a074-acb6bf6b46c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb" Oct 30 18:32:07.666553 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:07.666489 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62472024-eb2f-418e-a074-acb6bf6b46c6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-42b56365-tf5kb\" (UID: \"62472024-eb2f-418e-a074-acb6bf6b46c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb" Oct 30 18:32:07.683699 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:07.683677 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zjbkw\" (UniqueName: \"kubernetes.io/projected/62472024-eb2f-418e-a074-acb6bf6b46c6-kube-api-access-zjbkw\") pod \"maas-api-revoke-sub-42b56365-tf5kb\" (UID: \"62472024-eb2f-418e-a074-acb6bf6b46c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb" Oct 30 18:32:07.808975 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:07.808902 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb" Oct 30 18:32:08.107918 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:08.107886 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb"] Oct 30 18:32:08.109098 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:32:08.109075 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62472024_eb2f_418e_a074_acb6bf6b46c6.slice/crio-038ffa257db023ef2566b9bbe3e0b45e8fed65f0fbfeb7011a5f13c1a223c07a WatchSource:0}: Error finding container 038ffa257db023ef2566b9bbe3e0b45e8fed65f0fbfeb7011a5f13c1a223c07a: Status 404 returned error can't find the container with id 038ffa257db023ef2566b9bbe3e0b45e8fed65f0fbfeb7011a5f13c1a223c07a Oct 30 18:32:08.800889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:08.800828 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" podUID="0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9" containerName="main" containerID="cri-o://eccc08df0cc87f67fabc4d777c773dfb180f1e6d83c26e73238ad44c76abe095" gracePeriod=45 Oct 30 18:32:08.855353 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:08.855324 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb" event={"ID":"62472024-eb2f-418e-a074-acb6bf6b46c6","Type":"ContainerStarted","Data":"a4b6b1e0f023dbff944bfc3801ffa9b926498cf94ec14effeff1c6a26464feab"} Oct 30 18:32:08.855500 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:08.855378 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb" event={"ID":"62472024-eb2f-418e-a074-acb6bf6b46c6","Type":"ContainerStarted","Data":"038ffa257db023ef2566b9bbe3e0b45e8fed65f0fbfeb7011a5f13c1a223c07a"} Oct 30 18:32:09.112760 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.112735 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:32:09.157353 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.157301 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb" podStartSLOduration=2.157285322 podStartE2EDuration="2.157285322s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:08.916786808 +0000 UTC m=+1828.233207129" watchObservedRunningTime="2026-10-30 18:32:09.157285322 +0000 UTC m=+1828.473705643" Oct 30 18:32:09.184359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.184319 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-home\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-home\") pod \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " Oct 30 18:32:09.184589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.184384 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-tmp-dir\") pod \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " Oct 30 18:32:09.184589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.184424 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-tls-certs\" (UniqueName: \"kubernetes.io/secret/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-tls-certs\") pod \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " Oct 30 18:32:09.184589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.184536 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-dshm\") pod \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " Oct 30 18:32:09.184819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.184591 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-kube-api-access-h2nzc\" (UniqueName: \"kubernetes.io/projected/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-kube-api-access-h2nzc\") pod \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " Oct 30 18:32:09.184819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.184667 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-home" pod "0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9" (UID: "0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:09.184819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.184672 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-model-cache\") pod \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\" (UID: \"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9\") " Oct 30 18:32:09.184819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.184756 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-tmp-dir" pod "0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9" (UID: "0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:09.185048 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.184885 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-model-cache" pod "0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9" (UID: "0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:09.185377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.185265 5232 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-home\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:09.185482 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.185405 5232 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-tmp-dir\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:09.185482 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.185420 5232 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-model-cache\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:09.186993 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.186959 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-dshm" pod "0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9" (UID: "0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:09.186993 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.186981 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-tls-certs" pod "0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9" (UID: "0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:09.187162 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.187097 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-kube-api-access-h2nzc" pod "0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9" (UID: "0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9"). InnerVolumeSpecName "kube-api-access-h2nzc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:09.287143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.287109 5232 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-tls-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:09.287143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.287137 5232 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-dshm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:09.287143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.287148 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2nzc\" (UniqueName: \"kubernetes.io/projected/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9-kube-api-access-h2nzc\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:09.863189 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.863152 5232 generic.go:356] "Generic (PLEG): container finished" podID="62472024-eb2f-418e-a074-acb6bf6b46c6" containerID="a4b6b1e0f023dbff944bfc3801ffa9b926498cf94ec14effeff1c6a26464feab" exitCode=0 Oct 30 18:32:09.863356 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.863238 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb" event={"ID":"62472024-eb2f-418e-a074-acb6bf6b46c6","Type":"ContainerDied","Data":"a4b6b1e0f023dbff944bfc3801ffa9b926498cf94ec14effeff1c6a26464feab"} Oct 30 18:32:09.864638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.864617 5232 generic.go:356] "Generic (PLEG): container finished" podID="0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9" containerID="eccc08df0cc87f67fabc4d777c773dfb180f1e6d83c26e73238ad44c76abe095" exitCode=0 Oct 30 18:32:09.864728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.864653 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" event={"ID":"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9","Type":"ContainerDied","Data":"eccc08df0cc87f67fabc4d777c773dfb180f1e6d83c26e73238ad44c76abe095"} Oct 30 18:32:09.864728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.864662 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" Oct 30 18:32:09.864728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.864689 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" event={"ID":"0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9","Type":"ContainerDied","Data":"58f46d0a9183df387d38bb821333d09c88eedc8ac355a3a5e453b6dad18209d4"} Oct 30 18:32:09.864728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.864707 5232 scope.go:122] "RemoveContainer" containerID="eccc08df0cc87f67fabc4d777c773dfb180f1e6d83c26e73238ad44c76abe095" Oct 30 18:32:09.875416 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.875401 5232 scope.go:122] "RemoveContainer" containerID="eccc08df0cc87f67fabc4d777c773dfb180f1e6d83c26e73238ad44c76abe095" Oct 30 18:32:09.875806 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:32:09.875789 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eccc08df0cc87f67fabc4d777c773dfb180f1e6d83c26e73238ad44c76abe095\": container with ID starting with eccc08df0cc87f67fabc4d777c773dfb180f1e6d83c26e73238ad44c76abe095 not found: ID does not exist" containerID="eccc08df0cc87f67fabc4d777c773dfb180f1e6d83c26e73238ad44c76abe095" Oct 30 18:32:09.875853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.875814 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eccc08df0cc87f67fabc4d777c773dfb180f1e6d83c26e73238ad44c76abe095"} err="failed to get container status \"eccc08df0cc87f67fabc4d777c773dfb180f1e6d83c26e73238ad44c76abe095\": rpc error: code = NotFound desc = could not find container \"eccc08df0cc87f67fabc4d777c773dfb180f1e6d83c26e73238ad44c76abe095\": container with ID starting with eccc08df0cc87f67fabc4d777c773dfb180f1e6d83c26e73238ad44c76abe095 not found: ID does not exist" Oct 30 18:32:09.946129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.946096 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5"] Oct 30 18:32:09.955280 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:09.955240 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5"] Oct 30 18:32:10.103314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:10.103275 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-0f0c50/test-model-0f0c50-kserve-84cdc764c-bjpg5" podUID="0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9" containerName="main" probeResult="failure" output="Get \"https://10.133.0.220:8000/ready\": context deadline exceeded" Oct 30 18:32:10.804688 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:10.804652 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9" path="/var/lib/kubelet/pods/0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9/volumes" Oct 30 18:32:11.014236 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:11.014210 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb" Oct 30 18:32:11.120327 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:11.120244 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62472024-eb2f-418e-a074-acb6bf6b46c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62472024-eb2f-418e-a074-acb6bf6b46c6-maas-api-xks-ca\") pod \"62472024-eb2f-418e-a074-acb6bf6b46c6\" (UID: \"62472024-eb2f-418e-a074-acb6bf6b46c6\") " Oct 30 18:32:11.120327 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:11.120285 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62472024-eb2f-418e-a074-acb6bf6b46c6-kube-api-access-zjbkw\" (UniqueName: \"kubernetes.io/projected/62472024-eb2f-418e-a074-acb6bf6b46c6-kube-api-access-zjbkw\") pod \"62472024-eb2f-418e-a074-acb6bf6b46c6\" (UID: \"62472024-eb2f-418e-a074-acb6bf6b46c6\") " Oct 30 18:32:11.120628 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:11.120366 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62472024-eb2f-418e-a074-acb6bf6b46c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62472024-eb2f-418e-a074-acb6bf6b46c6-maas-api-openshift-ca\") pod \"62472024-eb2f-418e-a074-acb6bf6b46c6\" (UID: \"62472024-eb2f-418e-a074-acb6bf6b46c6\") " Oct 30 18:32:11.120806 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:11.120782 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62472024-eb2f-418e-a074-acb6bf6b46c6-maas-api-openshift-ca" pod "62472024-eb2f-418e-a074-acb6bf6b46c6" (UID: "62472024-eb2f-418e-a074-acb6bf6b46c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:11.122556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:11.122534 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62472024-eb2f-418e-a074-acb6bf6b46c6-maas-api-xks-ca" pod "62472024-eb2f-418e-a074-acb6bf6b46c6" (UID: "62472024-eb2f-418e-a074-acb6bf6b46c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:11.122631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:11.122553 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62472024-eb2f-418e-a074-acb6bf6b46c6-kube-api-access-zjbkw" pod "62472024-eb2f-418e-a074-acb6bf6b46c6" (UID: "62472024-eb2f-418e-a074-acb6bf6b46c6"). InnerVolumeSpecName "kube-api-access-zjbkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:11.222354 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:11.222318 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/62472024-eb2f-418e-a074-acb6bf6b46c6-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:11.222354 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:11.222348 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zjbkw\" (UniqueName: \"kubernetes.io/projected/62472024-eb2f-418e-a074-acb6bf6b46c6-kube-api-access-zjbkw\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:11.222354 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:11.222358 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/62472024-eb2f-418e-a074-acb6bf6b46c6-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:11.879790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:11.879754 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb" event={"ID":"62472024-eb2f-418e-a074-acb6bf6b46c6","Type":"ContainerDied","Data":"038ffa257db023ef2566b9bbe3e0b45e8fed65f0fbfeb7011a5f13c1a223c07a"} Oct 30 18:32:11.879790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:11.879778 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb" Oct 30 18:32:11.880257 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:11.879808 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="038ffa257db023ef2566b9bbe3e0b45e8fed65f0fbfeb7011a5f13c1a223c07a" Oct 30 18:32:18.045478 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:18.045444 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz"] Oct 30 18:32:18.085214 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:18.085181 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7"] Oct 30 18:32:19.579842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.579806 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc"] Oct 30 18:32:19.580369 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.580355 5232 cpu_manager.go:414] "containerMap: removing container" podUID="0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9" containerName="main" Oct 30 18:32:19.580409 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.580370 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9" containerName="main" Oct 30 18:32:19.580409 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.580379 5232 cpu_manager.go:414] "containerMap: removing container" podUID="62472024-eb2f-418e-a074-acb6bf6b46c6" containerName="revoke-keys" Oct 30 18:32:19.580409 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.580384 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62472024-eb2f-418e-a074-acb6bf6b46c6" containerName="revoke-keys" Oct 30 18:32:19.580554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.580483 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="0dba5d19-7ec3-4fdc-ba5f-4e9c695d70c9" containerName="main" Oct 30 18:32:19.580554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.580490 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="62472024-eb2f-418e-a074-acb6bf6b46c6" containerName="revoke-keys" Oct 30 18:32:19.584028 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.584010 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc" Oct 30 18:32:19.603537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.603486 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc"] Oct 30 18:32:19.612438 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.612405 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:32:19.612608 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.612589 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:32:19.729377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.729335 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7jlmn\" (UniqueName: \"kubernetes.io/projected/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-kube-api-access-7jlmn\") pod \"maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc\" (UID: \"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc" Oct 30 18:32:19.729614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.729400 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc\" (UID: \"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc" Oct 30 18:32:19.729710 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.729689 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc\" (UID: \"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc" Oct 30 18:32:19.830565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.830451 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7jlmn\" (UniqueName: \"kubernetes.io/projected/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-kube-api-access-7jlmn\") pod \"maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc\" (UID: \"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc" Oct 30 18:32:19.830565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.830533 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc\" (UID: \"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc" Oct 30 18:32:19.830819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.830721 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc\" (UID: \"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc" Oct 30 18:32:19.831367 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.831339 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc\" (UID: \"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc" Oct 30 18:32:19.833044 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.833026 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc\" (UID: \"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc" Oct 30 18:32:19.850024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.849994 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7jlmn\" (UniqueName: \"kubernetes.io/projected/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-kube-api-access-7jlmn\") pod \"maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc\" (UID: \"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc" Oct 30 18:32:19.897123 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:19.897084 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc" Oct 30 18:32:20.200332 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:20.200301 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc"] Oct 30 18:32:20.210632 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:32:20.202069 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9df3fd4c_5dfd_4df8_a9fa_79ea6b8aabae.slice/crio-c0f086a17cbd9dde0392f9b08041ff2e1dd2313fc98a13fc94f8e6bff29bb945 WatchSource:0}: Error finding container c0f086a17cbd9dde0392f9b08041ff2e1dd2313fc98a13fc94f8e6bff29bb945: Status 404 returned error can't find the container with id c0f086a17cbd9dde0392f9b08041ff2e1dd2313fc98a13fc94f8e6bff29bb945 Oct 30 18:32:20.942360 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:20.942265 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc" event={"ID":"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae","Type":"ContainerStarted","Data":"6625673d980fdedfe55766768693171e22a17794246e08d14c39d919ca7bf359"} Oct 30 18:32:20.942360 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:20.942327 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc" event={"ID":"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae","Type":"ContainerStarted","Data":"c0f086a17cbd9dde0392f9b08041ff2e1dd2313fc98a13fc94f8e6bff29bb945"} Oct 30 18:32:20.982980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:20.982908 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc" podStartSLOduration=1.982889588 podStartE2EDuration="1.982889588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:20.981536896 +0000 UTC m=+1840.297957212" watchObservedRunningTime="2026-10-30 18:32:20.982889588 +0000 UTC m=+1840.299309913" Oct 30 18:32:21.918783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:21.918735 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" podUID="3a723eea-87a6-48d9-b2cd-a2df079841c6" containerName="main" containerID="cri-o://330ac5a549fdf4842a4923e935c72fe05c20de44e3b3f3d980f5d3af5aeae5b3" gracePeriod=45 Oct 30 18:32:21.943596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:21.943565 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" podUID="3a723eea-87a6-48d9-b2cd-a2df079841c6" containerName="main" probeResult="failure" output="Get \"https://10.133.0.231:8000/ready\": dial tcp 10.133.0.231:8000: connect: connection refused" Oct 30 18:32:21.951369 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:21.951343 5232 generic.go:356] "Generic (PLEG): container finished" podID="9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae" containerID="6625673d980fdedfe55766768693171e22a17794246e08d14c39d919ca7bf359" exitCode=0 Oct 30 18:32:21.951518 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:21.951423 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc" event={"ID":"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae","Type":"ContainerDied","Data":"6625673d980fdedfe55766768693171e22a17794246e08d14c39d919ca7bf359"} Oct 30 18:32:22.245886 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.245852 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:32:22.367864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.367827 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-tmp-dir\") pod \"3a723eea-87a6-48d9-b2cd-a2df079841c6\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " Oct 30 18:32:22.368080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.367880 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-model-cache\") pod \"3a723eea-87a6-48d9-b2cd-a2df079841c6\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " Oct 30 18:32:22.368080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.367933 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-dshm\") pod \"3a723eea-87a6-48d9-b2cd-a2df079841c6\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " Oct 30 18:32:22.368080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.367973 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-home\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-home\") pod \"3a723eea-87a6-48d9-b2cd-a2df079841c6\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " Oct 30 18:32:22.368273 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.368112 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a723eea-87a6-48d9-b2cd-a2df079841c6-kube-api-access-hfcjp\" (UniqueName: \"kubernetes.io/projected/3a723eea-87a6-48d9-b2cd-a2df079841c6-kube-api-access-hfcjp\") pod \"3a723eea-87a6-48d9-b2cd-a2df079841c6\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " Oct 30 18:32:22.368273 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.368115 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-tmp-dir" pod "3a723eea-87a6-48d9-b2cd-a2df079841c6" (UID: "3a723eea-87a6-48d9-b2cd-a2df079841c6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:22.368273 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.368155 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a723eea-87a6-48d9-b2cd-a2df079841c6-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a723eea-87a6-48d9-b2cd-a2df079841c6-tls-certs\") pod \"3a723eea-87a6-48d9-b2cd-a2df079841c6\" (UID: \"3a723eea-87a6-48d9-b2cd-a2df079841c6\") " Oct 30 18:32:22.368273 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.368227 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-model-cache" pod "3a723eea-87a6-48d9-b2cd-a2df079841c6" (UID: "3a723eea-87a6-48d9-b2cd-a2df079841c6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:22.368273 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.368259 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-home" pod "3a723eea-87a6-48d9-b2cd-a2df079841c6" (UID: "3a723eea-87a6-48d9-b2cd-a2df079841c6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:22.368567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.368549 5232 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-tmp-dir\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:22.368616 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.368572 5232 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-model-cache\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:22.368616 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.368584 5232 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-home\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:22.370460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.370436 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-dshm" pod "3a723eea-87a6-48d9-b2cd-a2df079841c6" (UID: "3a723eea-87a6-48d9-b2cd-a2df079841c6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:22.370585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.370480 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a723eea-87a6-48d9-b2cd-a2df079841c6-tls-certs" pod "3a723eea-87a6-48d9-b2cd-a2df079841c6" (UID: "3a723eea-87a6-48d9-b2cd-a2df079841c6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:22.370585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.370540 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a723eea-87a6-48d9-b2cd-a2df079841c6-kube-api-access-hfcjp" pod "3a723eea-87a6-48d9-b2cd-a2df079841c6" (UID: "3a723eea-87a6-48d9-b2cd-a2df079841c6"). InnerVolumeSpecName "kube-api-access-hfcjp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:22.469618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.469539 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfcjp\" (UniqueName: \"kubernetes.io/projected/3a723eea-87a6-48d9-b2cd-a2df079841c6-kube-api-access-hfcjp\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:22.469618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.469566 5232 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a723eea-87a6-48d9-b2cd-a2df079841c6-tls-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:22.469618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.469576 5232 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a723eea-87a6-48d9-b2cd-a2df079841c6-dshm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:22.959772 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.959729 5232 generic.go:356] "Generic (PLEG): container finished" podID="3a723eea-87a6-48d9-b2cd-a2df079841c6" containerID="330ac5a549fdf4842a4923e935c72fe05c20de44e3b3f3d980f5d3af5aeae5b3" exitCode=0 Oct 30 18:32:22.960276 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.959799 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" Oct 30 18:32:22.960276 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.959819 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" event={"ID":"3a723eea-87a6-48d9-b2cd-a2df079841c6","Type":"ContainerDied","Data":"330ac5a549fdf4842a4923e935c72fe05c20de44e3b3f3d980f5d3af5aeae5b3"} Oct 30 18:32:22.960276 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.959871 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv" event={"ID":"3a723eea-87a6-48d9-b2cd-a2df079841c6","Type":"ContainerDied","Data":"8b7bc602141cb4c91ab0130a4884ce514f2447ba0be07b487df3ece67de7de69"} Oct 30 18:32:22.960276 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.959891 5232 scope.go:122] "RemoveContainer" containerID="330ac5a549fdf4842a4923e935c72fe05c20de44e3b3f3d980f5d3af5aeae5b3" Oct 30 18:32:22.969934 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.969910 5232 scope.go:122] "RemoveContainer" containerID="330ac5a549fdf4842a4923e935c72fe05c20de44e3b3f3d980f5d3af5aeae5b3" Oct 30 18:32:22.970241 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:32:22.970218 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"330ac5a549fdf4842a4923e935c72fe05c20de44e3b3f3d980f5d3af5aeae5b3\": container with ID starting with 330ac5a549fdf4842a4923e935c72fe05c20de44e3b3f3d980f5d3af5aeae5b3 not found: ID does not exist" containerID="330ac5a549fdf4842a4923e935c72fe05c20de44e3b3f3d980f5d3af5aeae5b3" Oct 30 18:32:22.970320 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:22.970255 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"330ac5a549fdf4842a4923e935c72fe05c20de44e3b3f3d980f5d3af5aeae5b3"} err="failed to get container status \"330ac5a549fdf4842a4923e935c72fe05c20de44e3b3f3d980f5d3af5aeae5b3\": rpc error: code = NotFound desc = could not find container \"330ac5a549fdf4842a4923e935c72fe05c20de44e3b3f3d980f5d3af5aeae5b3\": container with ID starting with 330ac5a549fdf4842a4923e935c72fe05c20de44e3b3f3d980f5d3af5aeae5b3 not found: ID does not exist" Oct 30 18:32:23.010626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:23.010594 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv"] Oct 30 18:32:23.024687 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:23.024656 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-1b0848/test-model-1b0848-kserve-648bb66d45-xf6tv"] Oct 30 18:32:23.046701 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:23.046655 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" podUID="f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b" containerName="payload-processing" containerID="cri-o://cae053a99026cc0ed6be477cb7ece43788a46b1187c436d947539cdfee1e33e1" gracePeriod=25 Oct 30 18:32:23.085955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:23.085920 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" podUID="29af1a00-c1ac-401e-abcd-eefeaafab0de" containerName="payload-pre-processing" containerID="cri-o://336083d06a070e03fd45ca69dea9d416af352630236c85cf0127d784e060d077" gracePeriod=25 Oct 30 18:32:23.099666 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:23.099643 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc" Oct 30 18:32:23.279060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:23.279031 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-kube-api-access-7jlmn\" (UniqueName: \"kubernetes.io/projected/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-kube-api-access-7jlmn\") pod \"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae\" (UID: \"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae\") " Oct 30 18:32:23.279271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:23.279117 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-maas-api-openshift-ca\") pod \"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae\" (UID: \"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae\") " Oct 30 18:32:23.279271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:23.279163 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-maas-api-xks-ca\") pod \"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae\" (UID: \"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae\") " Oct 30 18:32:23.279456 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:23.279430 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-maas-api-openshift-ca" pod "9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae" (UID: "9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:23.279604 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:23.279581 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:23.281424 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:23.281401 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-maas-api-xks-ca" pod "9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae" (UID: "9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:23.281502 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:23.281423 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-kube-api-access-7jlmn" pod "9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae" (UID: "9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae"). InnerVolumeSpecName "kube-api-access-7jlmn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:23.380951 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:23.380867 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7jlmn\" (UniqueName: \"kubernetes.io/projected/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-kube-api-access-7jlmn\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:23.380951 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:23.380897 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:23.968953 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:23.968920 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc" Oct 30 18:32:23.968953 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:23.968949 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc" event={"ID":"9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae","Type":"ContainerDied","Data":"c0f086a17cbd9dde0392f9b08041ff2e1dd2313fc98a13fc94f8e6bff29bb945"} Oct 30 18:32:23.969388 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:23.968990 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c0f086a17cbd9dde0392f9b08041ff2e1dd2313fc98a13fc94f8e6bff29bb945" Oct 30 18:32:24.802564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:24.802527 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a723eea-87a6-48d9-b2cd-a2df079841c6" path="/var/lib/kubelet/pods/3a723eea-87a6-48d9-b2cd-a2df079841c6/volumes" Oct 30 18:32:27.047423 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:32:27.047381 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:32:30.009639 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:30.009608 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k"] Oct 30 18:32:31.503990 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:31.503957 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc"] Oct 30 18:32:31.515939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:31.515906 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-1b0848-x69tc"] Oct 30 18:32:31.601347 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:31.601303 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" podUID="29af1a00-c1ac-401e-abcd-eefeaafab0de" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:32:31.603269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:31.603224 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" podUID="f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:32:32.803336 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:32.803289 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae" path="/var/lib/kubelet/pods/9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae/volumes" Oct 30 18:32:34.277974 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:34.277935 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55"] Oct 30 18:32:34.278385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:34.278158 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" podUID="ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" containerName="istio-proxy" containerID="cri-o://e3c5bd05ecc786eef012e9a04c6197c94ea48748f81a7a89ab6a74820f34a2d7" gracePeriod=30 Oct 30 18:32:35.010672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:35.010626 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k" podUID="0f9721f2-b5b3-4ebb-8677-3e5f1192fe74" containerName="payload-processing" containerID="cri-o://cab48da4da62814999449d9cbf05c0f3e720f4fa46ca0b26e4cead20b76db908" gracePeriod=25 Oct 30 18:32:35.262719 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:35.262659 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k" Oct 30 18:32:35.339100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:35.339066 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f9721f2-b5b3-4ebb-8677-3e5f1192fe74-kube-api-access-vszj9\" (UniqueName: \"kubernetes.io/projected/0f9721f2-b5b3-4ebb-8677-3e5f1192fe74-kube-api-access-vszj9\") pod \"0f9721f2-b5b3-4ebb-8677-3e5f1192fe74\" (UID: \"0f9721f2-b5b3-4ebb-8677-3e5f1192fe74\") " Oct 30 18:32:35.339527 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:35.339161 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f9721f2-b5b3-4ebb-8677-3e5f1192fe74-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9721f2-b5b3-4ebb-8677-3e5f1192fe74-plugins-config-volume\") pod \"0f9721f2-b5b3-4ebb-8677-3e5f1192fe74\" (UID: \"0f9721f2-b5b3-4ebb-8677-3e5f1192fe74\") " Oct 30 18:32:35.339586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:35.339558 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f9721f2-b5b3-4ebb-8677-3e5f1192fe74-plugins-config-volume" pod "0f9721f2-b5b3-4ebb-8677-3e5f1192fe74" (UID: "0f9721f2-b5b3-4ebb-8677-3e5f1192fe74"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:35.341470 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:35.341446 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f9721f2-b5b3-4ebb-8677-3e5f1192fe74-kube-api-access-vszj9" pod "0f9721f2-b5b3-4ebb-8677-3e5f1192fe74" (UID: "0f9721f2-b5b3-4ebb-8677-3e5f1192fe74"). InnerVolumeSpecName "kube-api-access-vszj9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:35.440673 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:35.440642 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0f9721f2-b5b3-4ebb-8677-3e5f1192fe74-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:35.440673 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:35.440669 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vszj9\" (UniqueName: \"kubernetes.io/projected/0f9721f2-b5b3-4ebb-8677-3e5f1192fe74-kube-api-access-vszj9\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:36.065151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:36.065114 5232 generic.go:356] "Generic (PLEG): container finished" podID="0f9721f2-b5b3-4ebb-8677-3e5f1192fe74" containerID="cab48da4da62814999449d9cbf05c0f3e720f4fa46ca0b26e4cead20b76db908" exitCode=0 Oct 30 18:32:36.065351 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:36.065164 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k" Oct 30 18:32:36.065351 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:36.065192 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k" event={"ID":"0f9721f2-b5b3-4ebb-8677-3e5f1192fe74","Type":"ContainerDied","Data":"cab48da4da62814999449d9cbf05c0f3e720f4fa46ca0b26e4cead20b76db908"} Oct 30 18:32:36.065351 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:36.065243 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k" event={"ID":"0f9721f2-b5b3-4ebb-8677-3e5f1192fe74","Type":"ContainerDied","Data":"8b04e11c0febce3dd25731a5e789520ee3ccf48689b632c5f1df9891a3324c8d"} Oct 30 18:32:36.065351 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:36.065266 5232 scope.go:122] "RemoveContainer" containerID="cab48da4da62814999449d9cbf05c0f3e720f4fa46ca0b26e4cead20b76db908" Oct 30 18:32:36.075948 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:36.075929 5232 scope.go:122] "RemoveContainer" containerID="cab48da4da62814999449d9cbf05c0f3e720f4fa46ca0b26e4cead20b76db908" Oct 30 18:32:36.076244 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:32:36.076222 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cab48da4da62814999449d9cbf05c0f3e720f4fa46ca0b26e4cead20b76db908\": container with ID starting with cab48da4da62814999449d9cbf05c0f3e720f4fa46ca0b26e4cead20b76db908 not found: ID does not exist" containerID="cab48da4da62814999449d9cbf05c0f3e720f4fa46ca0b26e4cead20b76db908" Oct 30 18:32:36.076301 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:36.076253 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cab48da4da62814999449d9cbf05c0f3e720f4fa46ca0b26e4cead20b76db908"} err="failed to get container status \"cab48da4da62814999449d9cbf05c0f3e720f4fa46ca0b26e4cead20b76db908\": rpc error: code = NotFound desc = could not find container \"cab48da4da62814999449d9cbf05c0f3e720f4fa46ca0b26e4cead20b76db908\": container with ID starting with cab48da4da62814999449d9cbf05c0f3e720f4fa46ca0b26e4cead20b76db908 not found: ID does not exist" Oct 30 18:32:36.138956 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:36.138918 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k"] Oct 30 18:32:36.150953 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:36.150925 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-1b0848-79644c86fd-6ss2k"] Oct 30 18:32:36.802283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:36.802246 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f9721f2-b5b3-4ebb-8677-3e5f1192fe74" path="/var/lib/kubelet/pods/0f9721f2-b5b3-4ebb-8677-3e5f1192fe74/volumes" Oct 30 18:32:39.302958 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.302929 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm"] Oct 30 18:32:39.303390 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.303142 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" podUID="de317989-41ca-479f-bcc1-9c7b5a166e70" containerName="istio-proxy" containerID="cri-o://381dd6b472d3c1be4ad5a54e72486222ba4ab85cf75b4ba876952f77f1e138b8" gracePeriod=30 Oct 30 18:32:39.613448 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.613418 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:32:39.700330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.700301 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-workload-certs\") pod \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " Oct 30 18:32:39.700497 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.700359 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-token\" (UniqueName: \"kubernetes.io/projected/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-token\") pod \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " Oct 30 18:32:39.700497 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.700395 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-credential-socket\") pod \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " Oct 30 18:32:39.700497 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.700431 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istiod-ca-cert\") pod \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " Oct 30 18:32:39.700497 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.700483 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-envoy\") pod \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " Oct 30 18:32:39.700728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.700525 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-workload-socket\") pod \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " Oct 30 18:32:39.700728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.700620 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-data\") pod \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " Oct 30 18:32:39.700728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.700699 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-podinfo\") pod \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " Oct 30 18:32:39.700892 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.700751 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-kube-api-access-v4tnc\" (UniqueName: \"kubernetes.io/projected/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-kube-api-access-v4tnc\") pod \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\" (UID: \"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac\") " Oct 30 18:32:39.700980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.700660 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-workload-certs" pod "ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" (UID: "ae969ba8-cd15-49e8-b5d8-64861e4ce6ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:39.701042 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.700775 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-credential-socket" pod "ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" (UID: "ae969ba8-cd15-49e8-b5d8-64861e4ce6ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:39.701042 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.700974 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istiod-ca-cert" pod "ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" (UID: "ae969ba8-cd15-49e8-b5d8-64861e4ce6ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:39.701042 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.700871 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-workload-socket" pod "ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" (UID: "ae969ba8-cd15-49e8-b5d8-64861e4ce6ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:39.701042 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.700934 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-data" pod "ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" (UID: "ae969ba8-cd15-49e8-b5d8-64861e4ce6ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:39.701295 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.701271 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:39.701359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.701300 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:39.701359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.701317 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:39.701359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.701331 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:39.701359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.701343 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:39.702908 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.702881 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-token" pod "ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" (UID: "ae969ba8-cd15-49e8-b5d8-64861e4ce6ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:39.703143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.703127 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-kube-api-access-v4tnc" pod "ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" (UID: "ae969ba8-cd15-49e8-b5d8-64861e4ce6ac"). InnerVolumeSpecName "kube-api-access-v4tnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:39.703421 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.703406 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-podinfo" pod "ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" (UID: "ae969ba8-cd15-49e8-b5d8-64861e4ce6ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:32:39.703471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.703430 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-envoy" pod "ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" (UID: "ae969ba8-cd15-49e8-b5d8-64861e4ce6ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:39.802621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.802584 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:39.802621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.802612 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:39.802621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.802621 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v4tnc\" (UniqueName: \"kubernetes.io/projected/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-kube-api-access-v4tnc\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:39.802621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.802630 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:39.876398 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.876305 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp"] Oct 30 18:32:39.876881 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.876863 5232 cpu_manager.go:414] "containerMap: removing container" podUID="3a723eea-87a6-48d9-b2cd-a2df079841c6" containerName="main" Oct 30 18:32:39.876881 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.876880 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a723eea-87a6-48d9-b2cd-a2df079841c6" containerName="main" Oct 30 18:32:39.877068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.876909 5232 cpu_manager.go:414] "containerMap: removing container" podUID="ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" containerName="istio-proxy" Oct 30 18:32:39.877068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.876917 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" containerName="istio-proxy" Oct 30 18:32:39.877068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.876934 5232 cpu_manager.go:414] "containerMap: removing container" podUID="0f9721f2-b5b3-4ebb-8677-3e5f1192fe74" containerName="payload-processing" Oct 30 18:32:39.877068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.876940 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f9721f2-b5b3-4ebb-8677-3e5f1192fe74" containerName="payload-processing" Oct 30 18:32:39.877068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.876961 5232 cpu_manager.go:414] "containerMap: removing container" podUID="9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae" containerName="revoke-keys" Oct 30 18:32:39.877068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.876966 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae" containerName="revoke-keys" Oct 30 18:32:39.877068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.877041 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" containerName="istio-proxy" Oct 30 18:32:39.877068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.877049 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="9df3fd4c-5dfd-4df8-a9fa-79ea6b8aabae" containerName="revoke-keys" Oct 30 18:32:39.877068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.877057 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a723eea-87a6-48d9-b2cd-a2df079841c6" containerName="main" Oct 30 18:32:39.877068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.877068 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f9721f2-b5b3-4ebb-8677-3e5f1192fe74" containerName="payload-processing" Oct 30 18:32:39.880604 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.880578 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp" Oct 30 18:32:39.901080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.901052 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp"] Oct 30 18:32:39.962360 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.962332 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:32:39.962556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:39.962463 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:32:40.005117 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.005091 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326b98c0-32e4-492f-a132-2d92eb4c37f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp\" (UID: \"326b98c0-32e4-492f-a132-2d92eb4c37f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp" Oct 30 18:32:40.005117 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.005130 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qmc9j\" (UniqueName: \"kubernetes.io/projected/326b98c0-32e4-492f-a132-2d92eb4c37f9-kube-api-access-qmc9j\") pod \"maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp\" (UID: \"326b98c0-32e4-492f-a132-2d92eb4c37f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp" Oct 30 18:32:40.005351 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.005169 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326b98c0-32e4-492f-a132-2d92eb4c37f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp\" (UID: \"326b98c0-32e4-492f-a132-2d92eb4c37f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp" Oct 30 18:32:40.095107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.095074 5232 generic.go:356] "Generic (PLEG): container finished" podID="ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" containerID="e3c5bd05ecc786eef012e9a04c6197c94ea48748f81a7a89ab6a74820f34a2d7" exitCode=0 Oct 30 18:32:40.095314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.095136 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" Oct 30 18:32:40.095314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.095178 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" event={"ID":"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac","Type":"ContainerDied","Data":"e3c5bd05ecc786eef012e9a04c6197c94ea48748f81a7a89ab6a74820f34a2d7"} Oct 30 18:32:40.095314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.095242 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55" event={"ID":"ae969ba8-cd15-49e8-b5d8-64861e4ce6ac","Type":"ContainerDied","Data":"cc6e9d144a7dbd27bdaea73425ddb73d201451a3b7b430be228993ae78c157f3"} Oct 30 18:32:40.095314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.095266 5232 scope.go:122] "RemoveContainer" containerID="e3c5bd05ecc786eef012e9a04c6197c94ea48748f81a7a89ab6a74820f34a2d7" Oct 30 18:32:40.106009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.105984 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326b98c0-32e4-492f-a132-2d92eb4c37f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp\" (UID: \"326b98c0-32e4-492f-a132-2d92eb4c37f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp" Oct 30 18:32:40.106153 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.106022 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qmc9j\" (UniqueName: \"kubernetes.io/projected/326b98c0-32e4-492f-a132-2d92eb4c37f9-kube-api-access-qmc9j\") pod \"maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp\" (UID: \"326b98c0-32e4-492f-a132-2d92eb4c37f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp" Oct 30 18:32:40.106153 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.106054 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326b98c0-32e4-492f-a132-2d92eb4c37f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp\" (UID: \"326b98c0-32e4-492f-a132-2d92eb4c37f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp" Oct 30 18:32:40.106684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.106660 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326b98c0-32e4-492f-a132-2d92eb4c37f9-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp\" (UID: \"326b98c0-32e4-492f-a132-2d92eb4c37f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp" Oct 30 18:32:40.107168 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.107147 5232 scope.go:122] "RemoveContainer" containerID="78057977b8c711a6e9d6bdf34ef949a035e90cf4113d1777a0d5f0e401cd9865" Oct 30 18:32:40.109914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.109887 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326b98c0-32e4-492f-a132-2d92eb4c37f9-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp\" (UID: \"326b98c0-32e4-492f-a132-2d92eb4c37f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp" Oct 30 18:32:40.118726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.118707 5232 scope.go:122] "RemoveContainer" containerID="e3c5bd05ecc786eef012e9a04c6197c94ea48748f81a7a89ab6a74820f34a2d7" Oct 30 18:32:40.119012 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:32:40.118991 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3c5bd05ecc786eef012e9a04c6197c94ea48748f81a7a89ab6a74820f34a2d7\": container with ID starting with e3c5bd05ecc786eef012e9a04c6197c94ea48748f81a7a89ab6a74820f34a2d7 not found: ID does not exist" containerID="e3c5bd05ecc786eef012e9a04c6197c94ea48748f81a7a89ab6a74820f34a2d7" Oct 30 18:32:40.119097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.119020 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3c5bd05ecc786eef012e9a04c6197c94ea48748f81a7a89ab6a74820f34a2d7"} err="failed to get container status \"e3c5bd05ecc786eef012e9a04c6197c94ea48748f81a7a89ab6a74820f34a2d7\": rpc error: code = NotFound desc = could not find container \"e3c5bd05ecc786eef012e9a04c6197c94ea48748f81a7a89ab6a74820f34a2d7\": container with ID starting with e3c5bd05ecc786eef012e9a04c6197c94ea48748f81a7a89ab6a74820f34a2d7 not found: ID does not exist" Oct 30 18:32:40.119097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.119040 5232 scope.go:122] "RemoveContainer" containerID="78057977b8c711a6e9d6bdf34ef949a035e90cf4113d1777a0d5f0e401cd9865" Oct 30 18:32:40.119297 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:32:40.119278 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78057977b8c711a6e9d6bdf34ef949a035e90cf4113d1777a0d5f0e401cd9865\": container with ID starting with 78057977b8c711a6e9d6bdf34ef949a035e90cf4113d1777a0d5f0e401cd9865 not found: ID does not exist" containerID="78057977b8c711a6e9d6bdf34ef949a035e90cf4113d1777a0d5f0e401cd9865" Oct 30 18:32:40.119353 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.119307 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78057977b8c711a6e9d6bdf34ef949a035e90cf4113d1777a0d5f0e401cd9865"} err="failed to get container status \"78057977b8c711a6e9d6bdf34ef949a035e90cf4113d1777a0d5f0e401cd9865\": rpc error: code = NotFound desc = could not find container \"78057977b8c711a6e9d6bdf34ef949a035e90cf4113d1777a0d5f0e401cd9865\": container with ID starting with 78057977b8c711a6e9d6bdf34ef949a035e90cf4113d1777a0d5f0e401cd9865 not found: ID does not exist" Oct 30 18:32:40.125863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.125839 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qmc9j\" (UniqueName: \"kubernetes.io/projected/326b98c0-32e4-492f-a132-2d92eb4c37f9-kube-api-access-qmc9j\") pod \"maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp\" (UID: \"326b98c0-32e4-492f-a132-2d92eb4c37f9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp" Oct 30 18:32:40.145920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.145872 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55"] Oct 30 18:32:40.148727 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.148707 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-0f0c50-openshift-default-6f96d756dc-n7h55"] Oct 30 18:32:40.197573 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.197542 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp" Oct 30 18:32:40.492805 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.492777 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp"] Oct 30 18:32:40.494109 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:32:40.494077 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod326b98c0_32e4_492f_a132_2d92eb4c37f9.slice/crio-eb575a81a70d78153ad72d12b3b1546e0347c894225be1060f0b1a53c544e2c6 WatchSource:0}: Error finding container eb575a81a70d78153ad72d12b3b1546e0347c894225be1060f0b1a53c544e2c6: Status 404 returned error can't find the container with id eb575a81a70d78153ad72d12b3b1546e0347c894225be1060f0b1a53c544e2c6 Oct 30 18:32:40.803080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:40.802996 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" path="/var/lib/kubelet/pods/ae969ba8-cd15-49e8-b5d8-64861e4ce6ac/volumes" Oct 30 18:32:41.104734 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:41.104644 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp" event={"ID":"326b98c0-32e4-492f-a132-2d92eb4c37f9","Type":"ContainerStarted","Data":"1c3442c0a201135f1ff6f9a7841a7369e05affdce5b82eddcafa0628566f692a"} Oct 30 18:32:41.104734 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:41.104701 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp" event={"ID":"326b98c0-32e4-492f-a132-2d92eb4c37f9","Type":"ContainerStarted","Data":"eb575a81a70d78153ad72d12b3b1546e0347c894225be1060f0b1a53c544e2c6"} Oct 30 18:32:41.144268 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:41.144218 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp" podStartSLOduration=2.144203297 podStartE2EDuration="2.144203297s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:32:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:32:41.142216163 +0000 UTC m=+1860.458636481" watchObservedRunningTime="2026-10-30 18:32:41.144203297 +0000 UTC m=+1860.460623618" Oct 30 18:32:41.599816 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:41.599779 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" podUID="29af1a00-c1ac-401e-abcd-eefeaafab0de" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:32:41.600224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:41.599791 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" podUID="f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:32:42.113282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:42.113252 5232 generic.go:356] "Generic (PLEG): container finished" podID="326b98c0-32e4-492f-a132-2d92eb4c37f9" containerID="1c3442c0a201135f1ff6f9a7841a7369e05affdce5b82eddcafa0628566f692a" exitCode=0 Oct 30 18:32:42.113442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:42.113341 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp" event={"ID":"326b98c0-32e4-492f-a132-2d92eb4c37f9","Type":"ContainerDied","Data":"1c3442c0a201135f1ff6f9a7841a7369e05affdce5b82eddcafa0628566f692a"} Oct 30 18:32:43.283793 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.283764 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp" Oct 30 18:32:43.352206 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.352108 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/326b98c0-32e4-492f-a132-2d92eb4c37f9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326b98c0-32e4-492f-a132-2d92eb4c37f9-maas-api-xks-ca\") pod \"326b98c0-32e4-492f-a132-2d92eb4c37f9\" (UID: \"326b98c0-32e4-492f-a132-2d92eb4c37f9\") " Oct 30 18:32:43.352381 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.352207 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/326b98c0-32e4-492f-a132-2d92eb4c37f9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326b98c0-32e4-492f-a132-2d92eb4c37f9-maas-api-openshift-ca\") pod \"326b98c0-32e4-492f-a132-2d92eb4c37f9\" (UID: \"326b98c0-32e4-492f-a132-2d92eb4c37f9\") " Oct 30 18:32:43.352381 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.352326 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/326b98c0-32e4-492f-a132-2d92eb4c37f9-kube-api-access-qmc9j\" (UniqueName: \"kubernetes.io/projected/326b98c0-32e4-492f-a132-2d92eb4c37f9-kube-api-access-qmc9j\") pod \"326b98c0-32e4-492f-a132-2d92eb4c37f9\" (UID: \"326b98c0-32e4-492f-a132-2d92eb4c37f9\") " Oct 30 18:32:43.352583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.352555 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/326b98c0-32e4-492f-a132-2d92eb4c37f9-maas-api-openshift-ca" pod "326b98c0-32e4-492f-a132-2d92eb4c37f9" (UID: "326b98c0-32e4-492f-a132-2d92eb4c37f9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:43.352855 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.352831 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/326b98c0-32e4-492f-a132-2d92eb4c37f9-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:43.354927 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.354901 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/326b98c0-32e4-492f-a132-2d92eb4c37f9-maas-api-xks-ca" pod "326b98c0-32e4-492f-a132-2d92eb4c37f9" (UID: "326b98c0-32e4-492f-a132-2d92eb4c37f9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:43.355132 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.355111 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/326b98c0-32e4-492f-a132-2d92eb4c37f9-kube-api-access-qmc9j" pod "326b98c0-32e4-492f-a132-2d92eb4c37f9" (UID: "326b98c0-32e4-492f-a132-2d92eb4c37f9"). InnerVolumeSpecName "kube-api-access-qmc9j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:43.374827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.374795 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" Oct 30 18:32:43.377621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.377605 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" Oct 30 18:32:43.453951 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.453914 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b-kube-api-access-hxrvx\" (UniqueName: \"kubernetes.io/projected/f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b-kube-api-access-hxrvx\") pod \"f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b\" (UID: \"f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b\") " Oct 30 18:32:43.454143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.453984 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/29af1a00-c1ac-401e-abcd-eefeaafab0de-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29af1a00-c1ac-401e-abcd-eefeaafab0de-plugins-config-volume\") pod \"29af1a00-c1ac-401e-abcd-eefeaafab0de\" (UID: \"29af1a00-c1ac-401e-abcd-eefeaafab0de\") " Oct 30 18:32:43.454143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.454040 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b-plugins-config-volume\") pod \"f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b\" (UID: \"f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b\") " Oct 30 18:32:43.454143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.454106 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/29af1a00-c1ac-401e-abcd-eefeaafab0de-kube-api-access-qz6fn\" (UniqueName: \"kubernetes.io/projected/29af1a00-c1ac-401e-abcd-eefeaafab0de-kube-api-access-qz6fn\") pod \"29af1a00-c1ac-401e-abcd-eefeaafab0de\" (UID: \"29af1a00-c1ac-401e-abcd-eefeaafab0de\") " Oct 30 18:32:43.454436 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.454417 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/326b98c0-32e4-492f-a132-2d92eb4c37f9-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:43.454436 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.454437 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qmc9j\" (UniqueName: \"kubernetes.io/projected/326b98c0-32e4-492f-a132-2d92eb4c37f9-kube-api-access-qmc9j\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:43.454642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.454493 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/29af1a00-c1ac-401e-abcd-eefeaafab0de-plugins-config-volume" pod "29af1a00-c1ac-401e-abcd-eefeaafab0de" (UID: "29af1a00-c1ac-401e-abcd-eefeaafab0de"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:43.454642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.454570 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b-plugins-config-volume" pod "f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b" (UID: "f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:43.456929 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.456902 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b-kube-api-access-hxrvx" pod "f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b" (UID: "f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b"). InnerVolumeSpecName "kube-api-access-hxrvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:43.457679 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.457654 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/29af1a00-c1ac-401e-abcd-eefeaafab0de-kube-api-access-qz6fn" pod "29af1a00-c1ac-401e-abcd-eefeaafab0de" (UID: "29af1a00-c1ac-401e-abcd-eefeaafab0de"). InnerVolumeSpecName "kube-api-access-qz6fn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:43.555559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.555524 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hxrvx\" (UniqueName: \"kubernetes.io/projected/f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b-kube-api-access-hxrvx\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:43.555559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.555554 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/29af1a00-c1ac-401e-abcd-eefeaafab0de-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:43.555559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.555564 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:43.555822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:43.555573 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qz6fn\" (UniqueName: \"kubernetes.io/projected/29af1a00-c1ac-401e-abcd-eefeaafab0de-kube-api-access-qz6fn\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:44.127798 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.127765 5232 generic.go:356] "Generic (PLEG): container finished" podID="f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b" containerID="cae053a99026cc0ed6be477cb7ece43788a46b1187c436d947539cdfee1e33e1" exitCode=0 Oct 30 18:32:44.127985 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.127814 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" Oct 30 18:32:44.127985 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.127849 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" event={"ID":"f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b","Type":"ContainerDied","Data":"cae053a99026cc0ed6be477cb7ece43788a46b1187c436d947539cdfee1e33e1"} Oct 30 18:32:44.127985 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.127903 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz" event={"ID":"f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b","Type":"ContainerDied","Data":"5b2c3c61b7facbcbd87c1084208f337768f5685c87ef307d5c1af5d7576984a6"} Oct 30 18:32:44.127985 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.127926 5232 scope.go:122] "RemoveContainer" containerID="cae053a99026cc0ed6be477cb7ece43788a46b1187c436d947539cdfee1e33e1" Oct 30 18:32:44.129348 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.129323 5232 generic.go:356] "Generic (PLEG): container finished" podID="29af1a00-c1ac-401e-abcd-eefeaafab0de" containerID="336083d06a070e03fd45ca69dea9d416af352630236c85cf0127d784e060d077" exitCode=0 Oct 30 18:32:44.129451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.129365 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" Oct 30 18:32:44.129451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.129422 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" event={"ID":"29af1a00-c1ac-401e-abcd-eefeaafab0de","Type":"ContainerDied","Data":"336083d06a070e03fd45ca69dea9d416af352630236c85cf0127d784e060d077"} Oct 30 18:32:44.129599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.129473 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7" event={"ID":"29af1a00-c1ac-401e-abcd-eefeaafab0de","Type":"ContainerDied","Data":"5e79ee205835b2e027b1722b2583148dd54e978d047a7f99f3dfe89fe7af5a26"} Oct 30 18:32:44.131176 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.131154 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp" Oct 30 18:32:44.131300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.131224 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp" event={"ID":"326b98c0-32e4-492f-a132-2d92eb4c37f9","Type":"ContainerDied","Data":"eb575a81a70d78153ad72d12b3b1546e0347c894225be1060f0b1a53c544e2c6"} Oct 30 18:32:44.131300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.131266 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="eb575a81a70d78153ad72d12b3b1546e0347c894225be1060f0b1a53c544e2c6" Oct 30 18:32:44.140524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.140445 5232 scope.go:122] "RemoveContainer" containerID="cae053a99026cc0ed6be477cb7ece43788a46b1187c436d947539cdfee1e33e1" Oct 30 18:32:44.140810 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:32:44.140782 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cae053a99026cc0ed6be477cb7ece43788a46b1187c436d947539cdfee1e33e1\": container with ID starting with cae053a99026cc0ed6be477cb7ece43788a46b1187c436d947539cdfee1e33e1 not found: ID does not exist" containerID="cae053a99026cc0ed6be477cb7ece43788a46b1187c436d947539cdfee1e33e1" Oct 30 18:32:44.140914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.140818 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cae053a99026cc0ed6be477cb7ece43788a46b1187c436d947539cdfee1e33e1"} err="failed to get container status \"cae053a99026cc0ed6be477cb7ece43788a46b1187c436d947539cdfee1e33e1\": rpc error: code = NotFound desc = could not find container \"cae053a99026cc0ed6be477cb7ece43788a46b1187c436d947539cdfee1e33e1\": container with ID starting with cae053a99026cc0ed6be477cb7ece43788a46b1187c436d947539cdfee1e33e1 not found: ID does not exist" Oct 30 18:32:44.140914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.140841 5232 scope.go:122] "RemoveContainer" containerID="336083d06a070e03fd45ca69dea9d416af352630236c85cf0127d784e060d077" Oct 30 18:32:44.149987 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.149970 5232 scope.go:122] "RemoveContainer" containerID="336083d06a070e03fd45ca69dea9d416af352630236c85cf0127d784e060d077" Oct 30 18:32:44.150262 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:32:44.150244 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"336083d06a070e03fd45ca69dea9d416af352630236c85cf0127d784e060d077\": container with ID starting with 336083d06a070e03fd45ca69dea9d416af352630236c85cf0127d784e060d077 not found: ID does not exist" containerID="336083d06a070e03fd45ca69dea9d416af352630236c85cf0127d784e060d077" Oct 30 18:32:44.150329 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.150274 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"336083d06a070e03fd45ca69dea9d416af352630236c85cf0127d784e060d077"} err="failed to get container status \"336083d06a070e03fd45ca69dea9d416af352630236c85cf0127d784e060d077\": rpc error: code = NotFound desc = could not find container \"336083d06a070e03fd45ca69dea9d416af352630236c85cf0127d784e060d077\": container with ID starting with 336083d06a070e03fd45ca69dea9d416af352630236c85cf0127d784e060d077 not found: ID does not exist" Oct 30 18:32:44.162729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.162707 5232 patch_prober.go:29] interesting pod/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:32:44.162832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.162759 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" podUID="de317989-41ca-479f-bcc1-9c7b5a166e70" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:32:44.180068 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.180042 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7"] Oct 30 18:32:44.192882 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.192839 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-0f0c50-86786c47-8fsf7"] Oct 30 18:32:44.225167 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.225135 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz"] Oct 30 18:32:44.228523 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.228487 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-0f0c50-6f56666484-wxjfz"] Oct 30 18:32:44.564107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.564081 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:32:44.665940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.665860 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-podinfo\") pod \"de317989-41ca-479f-bcc1-9c7b5a166e70\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " Oct 30 18:32:44.666107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.665982 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-token\" (UniqueName: \"kubernetes.io/projected/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-token\") pod \"de317989-41ca-479f-bcc1-9c7b5a166e70\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " Oct 30 18:32:44.666107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.666030 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/de317989-41ca-479f-bcc1-9c7b5a166e70-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de317989-41ca-479f-bcc1-9c7b5a166e70-istiod-ca-cert\") pod \"de317989-41ca-479f-bcc1-9c7b5a166e70\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " Oct 30 18:32:44.666107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.666073 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-data\") pod \"de317989-41ca-479f-bcc1-9c7b5a166e70\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " Oct 30 18:32:44.666286 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.666117 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-workload-socket\") pod \"de317989-41ca-479f-bcc1-9c7b5a166e70\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " Oct 30 18:32:44.666286 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.666189 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-envoy\") pod \"de317989-41ca-479f-bcc1-9c7b5a166e70\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " Oct 30 18:32:44.666286 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.666260 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/de317989-41ca-479f-bcc1-9c7b5a166e70-kube-api-access-v8g6j\" (UniqueName: \"kubernetes.io/projected/de317989-41ca-479f-bcc1-9c7b5a166e70-kube-api-access-v8g6j\") pod \"de317989-41ca-479f-bcc1-9c7b5a166e70\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " Oct 30 18:32:44.666451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.666320 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-credential-socket\") pod \"de317989-41ca-479f-bcc1-9c7b5a166e70\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " Oct 30 18:32:44.666451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.666351 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-workload-certs\") pod \"de317989-41ca-479f-bcc1-9c7b5a166e70\" (UID: \"de317989-41ca-479f-bcc1-9c7b5a166e70\") " Oct 30 18:32:44.666451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.666419 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-data" pod "de317989-41ca-479f-bcc1-9c7b5a166e70" (UID: "de317989-41ca-479f-bcc1-9c7b5a166e70"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:44.666451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.666431 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-workload-socket" pod "de317989-41ca-479f-bcc1-9c7b5a166e70" (UID: "de317989-41ca-479f-bcc1-9c7b5a166e70"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:44.666715 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.666489 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/de317989-41ca-479f-bcc1-9c7b5a166e70-istiod-ca-cert" pod "de317989-41ca-479f-bcc1-9c7b5a166e70" (UID: "de317989-41ca-479f-bcc1-9c7b5a166e70"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:44.666777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.666711 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-credential-socket" pod "de317989-41ca-479f-bcc1-9c7b5a166e70" (UID: "de317989-41ca-479f-bcc1-9c7b5a166e70"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:44.666777 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.666730 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-workload-certs" pod "de317989-41ca-479f-bcc1-9c7b5a166e70" (UID: "de317989-41ca-479f-bcc1-9c7b5a166e70"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:44.667104 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.666929 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/de317989-41ca-479f-bcc1-9c7b5a166e70-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:44.667104 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.666951 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:44.667104 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.666964 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:44.667104 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.666994 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:44.667104 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.667004 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:44.668637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.668608 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-token" pod "de317989-41ca-479f-bcc1-9c7b5a166e70" (UID: "de317989-41ca-479f-bcc1-9c7b5a166e70"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:44.668805 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.668789 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-envoy" pod "de317989-41ca-479f-bcc1-9c7b5a166e70" (UID: "de317989-41ca-479f-bcc1-9c7b5a166e70"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:32:44.668870 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.668862 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-podinfo" pod "de317989-41ca-479f-bcc1-9c7b5a166e70" (UID: "de317989-41ca-479f-bcc1-9c7b5a166e70"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:32:44.669059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.669035 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/de317989-41ca-479f-bcc1-9c7b5a166e70-kube-api-access-v8g6j" pod "de317989-41ca-479f-bcc1-9c7b5a166e70" (UID: "de317989-41ca-479f-bcc1-9c7b5a166e70"). InnerVolumeSpecName "kube-api-access-v8g6j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:44.768385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.768349 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8g6j\" (UniqueName: \"kubernetes.io/projected/de317989-41ca-479f-bcc1-9c7b5a166e70-kube-api-access-v8g6j\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:44.768385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.768382 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:44.768385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.768392 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:44.768684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.768402 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/de317989-41ca-479f-bcc1-9c7b5a166e70-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:44.802608 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.802576 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="29af1a00-c1ac-401e-abcd-eefeaafab0de" path="/var/lib/kubelet/pods/29af1a00-c1ac-401e-abcd-eefeaafab0de/volumes" Oct 30 18:32:44.802994 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:44.802978 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b" path="/var/lib/kubelet/pods/f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b/volumes" Oct 30 18:32:45.141576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:45.141535 5232 generic.go:356] "Generic (PLEG): container finished" podID="de317989-41ca-479f-bcc1-9c7b5a166e70" containerID="381dd6b472d3c1be4ad5a54e72486222ba4ab85cf75b4ba876952f77f1e138b8" exitCode=0 Oct 30 18:32:45.141784 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:45.141595 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" Oct 30 18:32:45.141784 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:45.141611 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" event={"ID":"de317989-41ca-479f-bcc1-9c7b5a166e70","Type":"ContainerDied","Data":"381dd6b472d3c1be4ad5a54e72486222ba4ab85cf75b4ba876952f77f1e138b8"} Oct 30 18:32:45.141784 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:45.141661 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm" event={"ID":"de317989-41ca-479f-bcc1-9c7b5a166e70","Type":"ContainerDied","Data":"54f4b77719cf2516a24cd3cd6da4af0ea44d4b03b88df847a414dae560aa52f5"} Oct 30 18:32:45.141784 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:45.141687 5232 scope.go:122] "RemoveContainer" containerID="381dd6b472d3c1be4ad5a54e72486222ba4ab85cf75b4ba876952f77f1e138b8" Oct 30 18:32:45.151581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:45.151565 5232 scope.go:122] "RemoveContainer" containerID="23575259bdca43d9ceb0b2822da983b9abd29ebdf649da0ce863a0f7385a4b6a" Oct 30 18:32:45.161283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:45.161266 5232 scope.go:122] "RemoveContainer" containerID="381dd6b472d3c1be4ad5a54e72486222ba4ab85cf75b4ba876952f77f1e138b8" Oct 30 18:32:45.161563 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:32:45.161543 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"381dd6b472d3c1be4ad5a54e72486222ba4ab85cf75b4ba876952f77f1e138b8\": container with ID starting with 381dd6b472d3c1be4ad5a54e72486222ba4ab85cf75b4ba876952f77f1e138b8 not found: ID does not exist" containerID="381dd6b472d3c1be4ad5a54e72486222ba4ab85cf75b4ba876952f77f1e138b8" Oct 30 18:32:45.161669 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:45.161574 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"381dd6b472d3c1be4ad5a54e72486222ba4ab85cf75b4ba876952f77f1e138b8"} err="failed to get container status \"381dd6b472d3c1be4ad5a54e72486222ba4ab85cf75b4ba876952f77f1e138b8\": rpc error: code = NotFound desc = could not find container \"381dd6b472d3c1be4ad5a54e72486222ba4ab85cf75b4ba876952f77f1e138b8\": container with ID starting with 381dd6b472d3c1be4ad5a54e72486222ba4ab85cf75b4ba876952f77f1e138b8 not found: ID does not exist" Oct 30 18:32:45.161669 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:45.161598 5232 scope.go:122] "RemoveContainer" containerID="23575259bdca43d9ceb0b2822da983b9abd29ebdf649da0ce863a0f7385a4b6a" Oct 30 18:32:45.161861 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:32:45.161844 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"23575259bdca43d9ceb0b2822da983b9abd29ebdf649da0ce863a0f7385a4b6a\": container with ID starting with 23575259bdca43d9ceb0b2822da983b9abd29ebdf649da0ce863a0f7385a4b6a not found: ID does not exist" containerID="23575259bdca43d9ceb0b2822da983b9abd29ebdf649da0ce863a0f7385a4b6a" Oct 30 18:32:45.161906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:45.161868 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"23575259bdca43d9ceb0b2822da983b9abd29ebdf649da0ce863a0f7385a4b6a"} err="failed to get container status \"23575259bdca43d9ceb0b2822da983b9abd29ebdf649da0ce863a0f7385a4b6a\": rpc error: code = NotFound desc = could not find container \"23575259bdca43d9ceb0b2822da983b9abd29ebdf649da0ce863a0f7385a4b6a\": container with ID starting with 23575259bdca43d9ceb0b2822da983b9abd29ebdf649da0ce863a0f7385a4b6a not found: ID does not exist" Oct 30 18:32:45.193191 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:45.193165 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm"] Oct 30 18:32:45.205353 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:45.205324 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-1b0848-openshift-default-6874f48d45-gf6xm"] Oct 30 18:32:45.363031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:45.362999 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6"] Oct 30 18:32:45.396797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:45.396712 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr"] Oct 30 18:32:46.801812 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:46.801775 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="de317989-41ca-479f-bcc1-9c7b5a166e70" path="/var/lib/kubelet/pods/de317989-41ca-479f-bcc1-9c7b5a166e70/volumes" Oct 30 18:32:48.037795 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:48.037755 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn"] Oct 30 18:32:48.045354 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:48.045318 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3aa66234-f85dn"] Oct 30 18:32:48.801619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:48.801581 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cb32688-9676-4e30-8b67-61d6344b4d8d" path="/var/lib/kubelet/pods/6cb32688-9676-4e30-8b67-61d6344b4d8d/volumes" Oct 30 18:32:50.363799 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:50.363752 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6" podUID="52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76" containerName="payload-processing" containerID="cri-o://dbd4403a7273ca1f579da876c0d52e60374cda59fa29f8775328e76c3ff47cc7" gracePeriod=25 Oct 30 18:32:50.397445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:50.397414 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" podUID="061ebd64-fd21-4ff1-ae1e-27148f48aa3c" containerName="payload-pre-processing" containerID="cri-o://80e44c432d6101a22e83799067f1988c52addc860aabea506dd1ab62f34fbe0c" gracePeriod=25 Oct 30 18:32:50.639200 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:50.639165 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6" Oct 30 18:32:50.740337 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:50.740305 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76-plugins-config-volume\") pod \"52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76\" (UID: \"52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76\") " Oct 30 18:32:50.740534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:50.740444 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76-kube-api-access-dr4dm\" (UniqueName: \"kubernetes.io/projected/52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76-kube-api-access-dr4dm\") pod \"52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76\" (UID: \"52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76\") " Oct 30 18:32:50.740697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:50.740675 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76-plugins-config-volume" pod "52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76" (UID: "52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:50.740869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:50.740853 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:50.742842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:50.742812 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76-kube-api-access-dr4dm" pod "52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76" (UID: "52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76"). InnerVolumeSpecName "kube-api-access-dr4dm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:50.842094 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:50.842058 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dr4dm\" (UniqueName: \"kubernetes.io/projected/52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76-kube-api-access-dr4dm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:51.188663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:51.188628 5232 generic.go:356] "Generic (PLEG): container finished" podID="52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76" containerID="dbd4403a7273ca1f579da876c0d52e60374cda59fa29f8775328e76c3ff47cc7" exitCode=0 Oct 30 18:32:51.188824 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:51.188675 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6" Oct 30 18:32:51.188824 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:51.188711 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6" event={"ID":"52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76","Type":"ContainerDied","Data":"dbd4403a7273ca1f579da876c0d52e60374cda59fa29f8775328e76c3ff47cc7"} Oct 30 18:32:51.188824 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:51.188765 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6" event={"ID":"52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76","Type":"ContainerDied","Data":"0bd2c106c548e289c8edfe4b5a8378824967c8f039731bbc88da3bdcab08a5b0"} Oct 30 18:32:51.188824 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:51.188788 5232 scope.go:122] "RemoveContainer" containerID="dbd4403a7273ca1f579da876c0d52e60374cda59fa29f8775328e76c3ff47cc7" Oct 30 18:32:51.198837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:51.198815 5232 scope.go:122] "RemoveContainer" containerID="dbd4403a7273ca1f579da876c0d52e60374cda59fa29f8775328e76c3ff47cc7" Oct 30 18:32:51.199144 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:32:51.199111 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbd4403a7273ca1f579da876c0d52e60374cda59fa29f8775328e76c3ff47cc7\": container with ID starting with dbd4403a7273ca1f579da876c0d52e60374cda59fa29f8775328e76c3ff47cc7 not found: ID does not exist" containerID="dbd4403a7273ca1f579da876c0d52e60374cda59fa29f8775328e76c3ff47cc7" Oct 30 18:32:51.199251 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:51.199149 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbd4403a7273ca1f579da876c0d52e60374cda59fa29f8775328e76c3ff47cc7"} err="failed to get container status \"dbd4403a7273ca1f579da876c0d52e60374cda59fa29f8775328e76c3ff47cc7\": rpc error: code = NotFound desc = could not find container \"dbd4403a7273ca1f579da876c0d52e60374cda59fa29f8775328e76c3ff47cc7\": container with ID starting with dbd4403a7273ca1f579da876c0d52e60374cda59fa29f8775328e76c3ff47cc7 not found: ID does not exist" Oct 30 18:32:51.241123 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:51.241083 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6"] Oct 30 18:32:51.244935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:51.244908 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-69c44c-684b86fb7b-gt9x6"] Oct 30 18:32:52.414483 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:52.414439 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw"] Oct 30 18:32:52.415037 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:52.414705 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" podUID="8cfb4330-a715-4d7b-aa34-560dfbecf89c" containerName="maas-api" containerID="cri-o://85a10794fe20e128a1f4d3a833bd0adee7c81760f5bdfa32c875eaf6206409f6" gracePeriod=30 Oct 30 18:32:52.716970 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:52.716942 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" Oct 30 18:32:52.803193 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:52.803154 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76" path="/var/lib/kubelet/pods/52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76/volumes" Oct 30 18:32:52.887555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:52.884451 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cfb4330-a715-4d7b-aa34-560dfbecf89c-kube-api-access-wwzwm\" (UniqueName: \"kubernetes.io/projected/8cfb4330-a715-4d7b-aa34-560dfbecf89c-kube-api-access-wwzwm\") pod \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\" (UID: \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\") " Oct 30 18:32:52.887555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:52.884533 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cfb4330-a715-4d7b-aa34-560dfbecf89c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cfb4330-a715-4d7b-aa34-560dfbecf89c-maas-api-tls\") pod \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\" (UID: \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\") " Oct 30 18:32:52.887555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:52.884564 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cfb4330-a715-4d7b-aa34-560dfbecf89c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfb4330-a715-4d7b-aa34-560dfbecf89c-metrics-certs\") pod \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\" (UID: \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\") " Oct 30 18:32:52.887555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:52.884707 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cfb4330-a715-4d7b-aa34-560dfbecf89c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cfb4330-a715-4d7b-aa34-560dfbecf89c-openshift-service-ca\") pod \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\" (UID: \"8cfb4330-a715-4d7b-aa34-560dfbecf89c\") " Oct 30 18:32:52.889142 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:52.889106 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cfb4330-a715-4d7b-aa34-560dfbecf89c-openshift-service-ca" pod "8cfb4330-a715-4d7b-aa34-560dfbecf89c" (UID: "8cfb4330-a715-4d7b-aa34-560dfbecf89c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:32:52.892145 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:52.892111 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cfb4330-a715-4d7b-aa34-560dfbecf89c-kube-api-access-wwzwm" pod "8cfb4330-a715-4d7b-aa34-560dfbecf89c" (UID: "8cfb4330-a715-4d7b-aa34-560dfbecf89c"). InnerVolumeSpecName "kube-api-access-wwzwm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:32:52.910602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:52.910524 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cfb4330-a715-4d7b-aa34-560dfbecf89c-maas-api-tls" pod "8cfb4330-a715-4d7b-aa34-560dfbecf89c" (UID: "8cfb4330-a715-4d7b-aa34-560dfbecf89c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:52.916767 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:52.916717 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cfb4330-a715-4d7b-aa34-560dfbecf89c-metrics-certs" pod "8cfb4330-a715-4d7b-aa34-560dfbecf89c" (UID: "8cfb4330-a715-4d7b-aa34-560dfbecf89c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:32:52.985696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:52.985672 5232 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8cfb4330-a715-4d7b-aa34-560dfbecf89c-openshift-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:52.985696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:52.985698 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wwzwm\" (UniqueName: \"kubernetes.io/projected/8cfb4330-a715-4d7b-aa34-560dfbecf89c-kube-api-access-wwzwm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:52.985896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:52.985708 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8cfb4330-a715-4d7b-aa34-560dfbecf89c-maas-api-tls\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:52.985896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:52.985718 5232 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8cfb4330-a715-4d7b-aa34-560dfbecf89c-metrics-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:32:53.204073 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:53.204039 5232 generic.go:356] "Generic (PLEG): container finished" podID="8cfb4330-a715-4d7b-aa34-560dfbecf89c" containerID="85a10794fe20e128a1f4d3a833bd0adee7c81760f5bdfa32c875eaf6206409f6" exitCode=0 Oct 30 18:32:53.204253 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:53.204101 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" Oct 30 18:32:53.204253 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:53.204121 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" event={"ID":"8cfb4330-a715-4d7b-aa34-560dfbecf89c","Type":"ContainerDied","Data":"85a10794fe20e128a1f4d3a833bd0adee7c81760f5bdfa32c875eaf6206409f6"} Oct 30 18:32:53.204253 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:53.204181 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw" event={"ID":"8cfb4330-a715-4d7b-aa34-560dfbecf89c","Type":"ContainerDied","Data":"f994465b2f1e60397ab159b63cd874595b94b8b01fdcfd3ef1e8af067e124849"} Oct 30 18:32:53.204253 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:53.204212 5232 scope.go:122] "RemoveContainer" containerID="85a10794fe20e128a1f4d3a833bd0adee7c81760f5bdfa32c875eaf6206409f6" Oct 30 18:32:53.216143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:53.216117 5232 scope.go:122] "RemoveContainer" containerID="85a10794fe20e128a1f4d3a833bd0adee7c81760f5bdfa32c875eaf6206409f6" Oct 30 18:32:53.216458 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:32:53.216438 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85a10794fe20e128a1f4d3a833bd0adee7c81760f5bdfa32c875eaf6206409f6\": container with ID starting with 85a10794fe20e128a1f4d3a833bd0adee7c81760f5bdfa32c875eaf6206409f6 not found: ID does not exist" containerID="85a10794fe20e128a1f4d3a833bd0adee7c81760f5bdfa32c875eaf6206409f6" Oct 30 18:32:53.216542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:53.216467 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85a10794fe20e128a1f4d3a833bd0adee7c81760f5bdfa32c875eaf6206409f6"} err="failed to get container status \"85a10794fe20e128a1f4d3a833bd0adee7c81760f5bdfa32c875eaf6206409f6\": rpc error: code = NotFound desc = could not find container \"85a10794fe20e128a1f4d3a833bd0adee7c81760f5bdfa32c875eaf6206409f6\": container with ID starting with 85a10794fe20e128a1f4d3a833bd0adee7c81760f5bdfa32c875eaf6206409f6 not found: ID does not exist" Oct 30 18:32:53.255307 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:53.255227 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw"] Oct 30 18:32:53.259118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:53.259088 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-b-69c44c-794888dc68-wl9xw"] Oct 30 18:32:54.821477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:54.821447 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cfb4330-a715-4d7b-aa34-560dfbecf89c" path="/var/lib/kubelet/pods/8cfb4330-a715-4d7b-aa34-560dfbecf89c/volumes" Oct 30 18:32:54.883181 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:54.883147 5232 scope.go:122] "RemoveContainer" containerID="023b46fdc47acc056bea6340f41909a24c36219dc62239fb6bb5a145b5f1f74a" Oct 30 18:32:54.892906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:54.892881 5232 scope.go:122] "RemoveContainer" containerID="b5fd730e9b318de0b4e91bc809bb38ce3b0030f778059f6a095f198f7c7ca6e9" Oct 30 18:32:54.901388 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:54.901369 5232 scope.go:122] "RemoveContainer" containerID="c435642bd2fc955ee767af96807a3a72fdcbfd9d8392f8b63bf95b3eea7d4ac8" Oct 30 18:32:54.910203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:54.910186 5232 scope.go:122] "RemoveContainer" containerID="b3108f4369363edfbfc63e3d374aa715e2964f3136fbc186d7332aa165952405" Oct 30 18:32:54.918408 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:54.918386 5232 scope.go:122] "RemoveContainer" containerID="97ae47a976812e07a85a90f9e7d486e51dd7159689a4faffce6ec3c82632e0f6" Oct 30 18:32:56.392090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:56.391888 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" podUID="061ebd64-fd21-4ff1-ae1e-27148f48aa3c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:32:57.035104 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:32:57.035068 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:32:58.457269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:58.457233 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp"] Oct 30 18:32:58.469123 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:58.469093 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-c9a107-q88hp"] Oct 30 18:32:58.802662 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:32:58.802627 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="326b98c0-32e4-492f-a132-2d92eb4c37f9" path="/var/lib/kubelet/pods/326b98c0-32e4-492f-a132-2d92eb4c37f9/volumes" Oct 30 18:33:01.496706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.496674 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg"] Oct 30 18:33:01.497148 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497126 5232 cpu_manager.go:414] "containerMap: removing container" podUID="de317989-41ca-479f-bcc1-9c7b5a166e70" containerName="istio-proxy" Oct 30 18:33:01.497148 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497138 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de317989-41ca-479f-bcc1-9c7b5a166e70" containerName="istio-proxy" Oct 30 18:33:01.497148 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497149 5232 cpu_manager.go:414] "containerMap: removing container" podUID="8cfb4330-a715-4d7b-aa34-560dfbecf89c" containerName="maas-api" Oct 30 18:33:01.497252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497156 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cfb4330-a715-4d7b-aa34-560dfbecf89c" containerName="maas-api" Oct 30 18:33:01.497252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497168 5232 cpu_manager.go:414] "containerMap: removing container" podUID="ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" containerName="istio-proxy" Oct 30 18:33:01.497252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497173 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" containerName="istio-proxy" Oct 30 18:33:01.497252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497187 5232 cpu_manager.go:414] "containerMap: removing container" podUID="29af1a00-c1ac-401e-abcd-eefeaafab0de" containerName="payload-pre-processing" Oct 30 18:33:01.497252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497192 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="29af1a00-c1ac-401e-abcd-eefeaafab0de" containerName="payload-pre-processing" Oct 30 18:33:01.497252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497202 5232 cpu_manager.go:414] "containerMap: removing container" podUID="52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76" containerName="payload-processing" Oct 30 18:33:01.497252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497208 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76" containerName="payload-processing" Oct 30 18:33:01.497252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497221 5232 cpu_manager.go:414] "containerMap: removing container" podUID="f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b" containerName="payload-processing" Oct 30 18:33:01.497252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497226 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b" containerName="payload-processing" Oct 30 18:33:01.497252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497236 5232 cpu_manager.go:414] "containerMap: removing container" podUID="326b98c0-32e4-492f-a132-2d92eb4c37f9" containerName="revoke-keys" Oct 30 18:33:01.497252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497241 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="326b98c0-32e4-492f-a132-2d92eb4c37f9" containerName="revoke-keys" Oct 30 18:33:01.497887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497315 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="52cc5e6c-0ec8-4e76-bcc5-2cad8679bc76" containerName="payload-processing" Oct 30 18:33:01.497887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497327 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae969ba8-cd15-49e8-b5d8-64861e4ce6ac" containerName="istio-proxy" Oct 30 18:33:01.497887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497333 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="de317989-41ca-479f-bcc1-9c7b5a166e70" containerName="istio-proxy" Oct 30 18:33:01.497887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497339 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="f83cfef5-1eeb-4d0f-9cc8-bb9de0d38d8b" containerName="payload-processing" Oct 30 18:33:01.497887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497345 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="326b98c0-32e4-492f-a132-2d92eb4c37f9" containerName="revoke-keys" Oct 30 18:33:01.497887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497354 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cfb4330-a715-4d7b-aa34-560dfbecf89c" containerName="maas-api" Oct 30 18:33:01.497887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.497361 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="29af1a00-c1ac-401e-abcd-eefeaafab0de" containerName="payload-pre-processing" Oct 30 18:33:01.500557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.500539 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg" Oct 30 18:33:01.519257 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.519219 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg"] Oct 30 18:33:01.577185 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.577147 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:33:01.577185 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.577181 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:33:01.610801 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.610760 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e57987-d470-42b7-8547-be295cc129e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg\" (UID: \"45e57987-d470-42b7-8547-be295cc129e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg" Oct 30 18:33:01.610978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.610812 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e57987-d470-42b7-8547-be295cc129e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg\" (UID: \"45e57987-d470-42b7-8547-be295cc129e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg" Oct 30 18:33:01.610978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.610937 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7xd4\" (UniqueName: \"kubernetes.io/projected/45e57987-d470-42b7-8547-be295cc129e5-kube-api-access-z7xd4\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg\" (UID: \"45e57987-d470-42b7-8547-be295cc129e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg" Oct 30 18:33:01.712344 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.712309 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7xd4\" (UniqueName: \"kubernetes.io/projected/45e57987-d470-42b7-8547-be295cc129e5-kube-api-access-z7xd4\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg\" (UID: \"45e57987-d470-42b7-8547-be295cc129e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg" Oct 30 18:33:01.712569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.712408 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e57987-d470-42b7-8547-be295cc129e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg\" (UID: \"45e57987-d470-42b7-8547-be295cc129e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg" Oct 30 18:33:01.712569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.712432 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e57987-d470-42b7-8547-be295cc129e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg\" (UID: \"45e57987-d470-42b7-8547-be295cc129e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg" Oct 30 18:33:01.713165 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.713137 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e57987-d470-42b7-8547-be295cc129e5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg\" (UID: \"45e57987-d470-42b7-8547-be295cc129e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg" Oct 30 18:33:01.715057 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.715027 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e57987-d470-42b7-8547-be295cc129e5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg\" (UID: \"45e57987-d470-42b7-8547-be295cc129e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg" Oct 30 18:33:01.731713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.731687 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7xd4\" (UniqueName: \"kubernetes.io/projected/45e57987-d470-42b7-8547-be295cc129e5-kube-api-access-z7xd4\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg\" (UID: \"45e57987-d470-42b7-8547-be295cc129e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg" Oct 30 18:33:01.813863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:01.813783 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg" Oct 30 18:33:02.129224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:02.127401 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg"] Oct 30 18:33:02.268081 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:02.268034 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg" event={"ID":"45e57987-d470-42b7-8547-be295cc129e5","Type":"ContainerStarted","Data":"5d38ea177b9e4557eeca3114cac8565a5ed24d5606429c086aedf2fe38945d59"} Oct 30 18:33:03.275680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:03.275640 5232 generic.go:356] "Generic (PLEG): container finished" podID="45e57987-d470-42b7-8547-be295cc129e5" containerID="a50a7c20e45f9c7c1d559a1e690668c2549b34dfce1f77215973b239c78e4ffd" exitCode=0 Oct 30 18:33:03.276090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:03.275718 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg" event={"ID":"45e57987-d470-42b7-8547-be295cc129e5","Type":"ContainerDied","Data":"a50a7c20e45f9c7c1d559a1e690668c2549b34dfce1f77215973b239c78e4ffd"} Oct 30 18:33:04.418489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:04.418463 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg" Oct 30 18:33:04.556696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:04.556607 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/45e57987-d470-42b7-8547-be295cc129e5-kube-api-access-z7xd4\" (UniqueName: \"kubernetes.io/projected/45e57987-d470-42b7-8547-be295cc129e5-kube-api-access-z7xd4\") pod \"45e57987-d470-42b7-8547-be295cc129e5\" (UID: \"45e57987-d470-42b7-8547-be295cc129e5\") " Oct 30 18:33:04.556696 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:04.556688 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/45e57987-d470-42b7-8547-be295cc129e5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e57987-d470-42b7-8547-be295cc129e5-maas-api-openshift-ca\") pod \"45e57987-d470-42b7-8547-be295cc129e5\" (UID: \"45e57987-d470-42b7-8547-be295cc129e5\") " Oct 30 18:33:04.556948 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:04.556739 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/45e57987-d470-42b7-8547-be295cc129e5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e57987-d470-42b7-8547-be295cc129e5-maas-api-xks-ca\") pod \"45e57987-d470-42b7-8547-be295cc129e5\" (UID: \"45e57987-d470-42b7-8547-be295cc129e5\") " Oct 30 18:33:04.557043 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:04.557019 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/45e57987-d470-42b7-8547-be295cc129e5-maas-api-openshift-ca" pod "45e57987-d470-42b7-8547-be295cc129e5" (UID: "45e57987-d470-42b7-8547-be295cc129e5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:04.557255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:04.557232 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/45e57987-d470-42b7-8547-be295cc129e5-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:04.559048 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:04.559022 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/45e57987-d470-42b7-8547-be295cc129e5-maas-api-xks-ca" pod "45e57987-d470-42b7-8547-be295cc129e5" (UID: "45e57987-d470-42b7-8547-be295cc129e5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:04.559145 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:04.559054 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/45e57987-d470-42b7-8547-be295cc129e5-kube-api-access-z7xd4" pod "45e57987-d470-42b7-8547-be295cc129e5" (UID: "45e57987-d470-42b7-8547-be295cc129e5"). InnerVolumeSpecName "kube-api-access-z7xd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:04.658024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:04.657989 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7xd4\" (UniqueName: \"kubernetes.io/projected/45e57987-d470-42b7-8547-be295cc129e5-kube-api-access-z7xd4\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:04.658024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:04.658019 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/45e57987-d470-42b7-8547-be295cc129e5-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:05.290243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:05.290212 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg" Oct 30 18:33:05.290243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:05.290238 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg" event={"ID":"45e57987-d470-42b7-8547-be295cc129e5","Type":"ContainerDied","Data":"5d38ea177b9e4557eeca3114cac8565a5ed24d5606429c086aedf2fe38945d59"} Oct 30 18:33:05.290470 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:05.290286 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5d38ea177b9e4557eeca3114cac8565a5ed24d5606429c086aedf2fe38945d59" Oct 30 18:33:06.391263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.391224 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" podUID="061ebd64-fd21-4ff1-ae1e-27148f48aa3c" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:33:06.488981 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.488948 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s"] Oct 30 18:33:06.489435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.489423 5232 cpu_manager.go:414] "containerMap: removing container" podUID="45e57987-d470-42b7-8547-be295cc129e5" containerName="revoke-keys" Oct 30 18:33:06.489495 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.489437 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="45e57987-d470-42b7-8547-be295cc129e5" containerName="revoke-keys" Oct 30 18:33:06.489495 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.489458 5232 cpu_manager.go:414] "containerMap: removing container" podUID="de317989-41ca-479f-bcc1-9c7b5a166e70" containerName="istio-proxy" Oct 30 18:33:06.489495 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.489463 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="de317989-41ca-479f-bcc1-9c7b5a166e70" containerName="istio-proxy" Oct 30 18:33:06.489631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.489586 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="de317989-41ca-479f-bcc1-9c7b5a166e70" containerName="istio-proxy" Oct 30 18:33:06.489631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.489596 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="45e57987-d470-42b7-8547-be295cc129e5" containerName="revoke-keys" Oct 30 18:33:06.492959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.492939 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s" Oct 30 18:33:06.514301 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.514271 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s"] Oct 30 18:33:06.569745 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.569704 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:33:06.569934 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.569765 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:33:06.577349 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.577324 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf57d029-4ffe-44e4-b025-f28c296405b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s\" (UID: \"cf57d029-4ffe-44e4-b025-f28c296405b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s" Oct 30 18:33:06.577462 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.577371 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgl5v\" (UniqueName: \"kubernetes.io/projected/cf57d029-4ffe-44e4-b025-f28c296405b2-kube-api-access-cgl5v\") pod \"maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s\" (UID: \"cf57d029-4ffe-44e4-b025-f28c296405b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s" Oct 30 18:33:06.577547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.577502 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf57d029-4ffe-44e4-b025-f28c296405b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s\" (UID: \"cf57d029-4ffe-44e4-b025-f28c296405b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s" Oct 30 18:33:06.678575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.678472 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf57d029-4ffe-44e4-b025-f28c296405b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s\" (UID: \"cf57d029-4ffe-44e4-b025-f28c296405b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s" Oct 30 18:33:06.678765 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.678575 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgl5v\" (UniqueName: \"kubernetes.io/projected/cf57d029-4ffe-44e4-b025-f28c296405b2-kube-api-access-cgl5v\") pod \"maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s\" (UID: \"cf57d029-4ffe-44e4-b025-f28c296405b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s" Oct 30 18:33:06.678765 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.678627 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf57d029-4ffe-44e4-b025-f28c296405b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s\" (UID: \"cf57d029-4ffe-44e4-b025-f28c296405b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s" Oct 30 18:33:06.679222 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.679193 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf57d029-4ffe-44e4-b025-f28c296405b2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s\" (UID: \"cf57d029-4ffe-44e4-b025-f28c296405b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s" Oct 30 18:33:06.681384 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.681355 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf57d029-4ffe-44e4-b025-f28c296405b2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s\" (UID: \"cf57d029-4ffe-44e4-b025-f28c296405b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s" Oct 30 18:33:06.698820 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.698790 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgl5v\" (UniqueName: \"kubernetes.io/projected/cf57d029-4ffe-44e4-b025-f28c296405b2-kube-api-access-cgl5v\") pod \"maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s\" (UID: \"cf57d029-4ffe-44e4-b025-f28c296405b2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s" Oct 30 18:33:06.806384 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:06.806352 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s" Oct 30 18:33:07.100823 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:07.100796 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s"] Oct 30 18:33:07.102491 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:33:07.102460 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf57d029_4ffe_44e4_b025_f28c296405b2.slice/crio-64880eba581624e7168d319978b7ae81a5640a661fe4f3184b4c1353edcd37ec WatchSource:0}: Error finding container 64880eba581624e7168d319978b7ae81a5640a661fe4f3184b4c1353edcd37ec: Status 404 returned error can't find the container with id 64880eba581624e7168d319978b7ae81a5640a661fe4f3184b4c1353edcd37ec Oct 30 18:33:07.312355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:07.312316 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s" event={"ID":"cf57d029-4ffe-44e4-b025-f28c296405b2","Type":"ContainerStarted","Data":"64880eba581624e7168d319978b7ae81a5640a661fe4f3184b4c1353edcd37ec"} Oct 30 18:33:08.319595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:08.319560 5232 generic.go:356] "Generic (PLEG): container finished" podID="cf57d029-4ffe-44e4-b025-f28c296405b2" containerID="a17ecb82cc82b1edc1205260b8d297a7128c91e6b890218e25cac40ceb15d023" exitCode=0 Oct 30 18:33:08.320045 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:08.319654 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s" event={"ID":"cf57d029-4ffe-44e4-b025-f28c296405b2","Type":"ContainerDied","Data":"a17ecb82cc82b1edc1205260b8d297a7128c91e6b890218e25cac40ceb15d023"} Oct 30 18:33:09.497527 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:09.497487 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s" Oct 30 18:33:09.610721 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:09.610639 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf57d029-4ffe-44e4-b025-f28c296405b2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf57d029-4ffe-44e4-b025-f28c296405b2-maas-api-xks-ca\") pod \"cf57d029-4ffe-44e4-b025-f28c296405b2\" (UID: \"cf57d029-4ffe-44e4-b025-f28c296405b2\") " Oct 30 18:33:09.610721 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:09.610709 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf57d029-4ffe-44e4-b025-f28c296405b2-kube-api-access-cgl5v\" (UniqueName: \"kubernetes.io/projected/cf57d029-4ffe-44e4-b025-f28c296405b2-kube-api-access-cgl5v\") pod \"cf57d029-4ffe-44e4-b025-f28c296405b2\" (UID: \"cf57d029-4ffe-44e4-b025-f28c296405b2\") " Oct 30 18:33:09.610897 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:09.610842 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf57d029-4ffe-44e4-b025-f28c296405b2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf57d029-4ffe-44e4-b025-f28c296405b2-maas-api-openshift-ca\") pod \"cf57d029-4ffe-44e4-b025-f28c296405b2\" (UID: \"cf57d029-4ffe-44e4-b025-f28c296405b2\") " Oct 30 18:33:09.611224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:09.611202 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf57d029-4ffe-44e4-b025-f28c296405b2-maas-api-openshift-ca" pod "cf57d029-4ffe-44e4-b025-f28c296405b2" (UID: "cf57d029-4ffe-44e4-b025-f28c296405b2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:09.612986 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:09.612958 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf57d029-4ffe-44e4-b025-f28c296405b2-maas-api-xks-ca" pod "cf57d029-4ffe-44e4-b025-f28c296405b2" (UID: "cf57d029-4ffe-44e4-b025-f28c296405b2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:09.613104 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:09.613007 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf57d029-4ffe-44e4-b025-f28c296405b2-kube-api-access-cgl5v" pod "cf57d029-4ffe-44e4-b025-f28c296405b2" (UID: "cf57d029-4ffe-44e4-b025-f28c296405b2"). InnerVolumeSpecName "kube-api-access-cgl5v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:09.712380 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:09.712339 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgl5v\" (UniqueName: \"kubernetes.io/projected/cf57d029-4ffe-44e4-b025-f28c296405b2-kube-api-access-cgl5v\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:09.712380 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:09.712377 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf57d029-4ffe-44e4-b025-f28c296405b2-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:09.712620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:09.712392 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf57d029-4ffe-44e4-b025-f28c296405b2-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:10.333143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:10.333109 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s" event={"ID":"cf57d029-4ffe-44e4-b025-f28c296405b2","Type":"ContainerDied","Data":"64880eba581624e7168d319978b7ae81a5640a661fe4f3184b4c1353edcd37ec"} Oct 30 18:33:10.333143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:10.333128 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s" Oct 30 18:33:10.333363 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:10.333155 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="64880eba581624e7168d319978b7ae81a5640a661fe4f3184b4c1353edcd37ec" Oct 30 18:33:10.660539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:10.655944 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" Oct 30 18:33:10.725445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:10.725408 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/061ebd64-fd21-4ff1-ae1e-27148f48aa3c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061ebd64-fd21-4ff1-ae1e-27148f48aa3c-plugins-config-volume\") pod \"061ebd64-fd21-4ff1-ae1e-27148f48aa3c\" (UID: \"061ebd64-fd21-4ff1-ae1e-27148f48aa3c\") " Oct 30 18:33:10.725898 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:10.725860 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/061ebd64-fd21-4ff1-ae1e-27148f48aa3c-kube-api-access-lzksj\" (UniqueName: \"kubernetes.io/projected/061ebd64-fd21-4ff1-ae1e-27148f48aa3c-kube-api-access-lzksj\") pod \"061ebd64-fd21-4ff1-ae1e-27148f48aa3c\" (UID: \"061ebd64-fd21-4ff1-ae1e-27148f48aa3c\") " Oct 30 18:33:10.726031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:10.725965 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/061ebd64-fd21-4ff1-ae1e-27148f48aa3c-plugins-config-volume" pod "061ebd64-fd21-4ff1-ae1e-27148f48aa3c" (UID: "061ebd64-fd21-4ff1-ae1e-27148f48aa3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:10.726266 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:10.726247 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/061ebd64-fd21-4ff1-ae1e-27148f48aa3c-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:10.728619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:10.728594 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/061ebd64-fd21-4ff1-ae1e-27148f48aa3c-kube-api-access-lzksj" pod "061ebd64-fd21-4ff1-ae1e-27148f48aa3c" (UID: "061ebd64-fd21-4ff1-ae1e-27148f48aa3c"). InnerVolumeSpecName "kube-api-access-lzksj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:10.827728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:10.827700 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lzksj\" (UniqueName: \"kubernetes.io/projected/061ebd64-fd21-4ff1-ae1e-27148f48aa3c-kube-api-access-lzksj\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:11.225715 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:11.225681 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2"] Oct 30 18:33:11.254622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:11.254587 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6"] Oct 30 18:33:11.340913 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:11.340881 5232 generic.go:356] "Generic (PLEG): container finished" podID="061ebd64-fd21-4ff1-ae1e-27148f48aa3c" containerID="80e44c432d6101a22e83799067f1988c52addc860aabea506dd1ab62f34fbe0c" exitCode=0 Oct 30 18:33:11.341116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:11.340951 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" Oct 30 18:33:11.341116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:11.340972 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" event={"ID":"061ebd64-fd21-4ff1-ae1e-27148f48aa3c","Type":"ContainerDied","Data":"80e44c432d6101a22e83799067f1988c52addc860aabea506dd1ab62f34fbe0c"} Oct 30 18:33:11.341116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:11.341030 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr" event={"ID":"061ebd64-fd21-4ff1-ae1e-27148f48aa3c","Type":"ContainerDied","Data":"9abbf59f86a1a685e8fce04cc080bee849a7ae526d233b7b55e2c5f0f1bfe421"} Oct 30 18:33:11.341116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:11.341055 5232 scope.go:122] "RemoveContainer" containerID="80e44c432d6101a22e83799067f1988c52addc860aabea506dd1ab62f34fbe0c" Oct 30 18:33:11.351191 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:11.351172 5232 scope.go:122] "RemoveContainer" containerID="80e44c432d6101a22e83799067f1988c52addc860aabea506dd1ab62f34fbe0c" Oct 30 18:33:11.351469 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:33:11.351446 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80e44c432d6101a22e83799067f1988c52addc860aabea506dd1ab62f34fbe0c\": container with ID starting with 80e44c432d6101a22e83799067f1988c52addc860aabea506dd1ab62f34fbe0c not found: ID does not exist" containerID="80e44c432d6101a22e83799067f1988c52addc860aabea506dd1ab62f34fbe0c" Oct 30 18:33:11.351545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:11.351476 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80e44c432d6101a22e83799067f1988c52addc860aabea506dd1ab62f34fbe0c"} err="failed to get container status \"80e44c432d6101a22e83799067f1988c52addc860aabea506dd1ab62f34fbe0c\": rpc error: code = NotFound desc = could not find container \"80e44c432d6101a22e83799067f1988c52addc860aabea506dd1ab62f34fbe0c\": container with ID starting with 80e44c432d6101a22e83799067f1988c52addc860aabea506dd1ab62f34fbe0c not found: ID does not exist" Oct 30 18:33:11.393122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:11.393089 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr"] Oct 30 18:33:11.401692 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:11.401666 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-69c44c-544478dfdc-c4fwr"] Oct 30 18:33:11.957465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:11.957432 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6"] Oct 30 18:33:11.990946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:11.990914 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm"] Oct 30 18:33:12.804619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:12.804579 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="061ebd64-fd21-4ff1-ae1e-27148f48aa3c" path="/var/lib/kubelet/pods/061ebd64-fd21-4ff1-ae1e-27148f48aa3c/volumes" Oct 30 18:33:16.940182 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:16.940148 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7"] Oct 30 18:33:16.957910 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:16.957865 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6" podUID="7703d89f-2407-4aa1-b263-1de901a8162b" containerName="payload-processing" containerID="cri-o://eca962e46774c2c90c0be38cec888918908821719aae4bd861a922a10ae23ba5" gracePeriod=25 Oct 30 18:33:16.973491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:16.973340 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl"] Oct 30 18:33:16.991173 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:16.991143 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm" podUID="1053887b-8b36-407c-8427-af7207e989b6" containerName="payload-pre-processing" containerID="cri-o://dcbf7007f6177639bc92b41a727528c37fa3db5e12d7bff1b7d07d55a874ffde" gracePeriod=25 Oct 30 18:33:17.261987 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.261963 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6" Oct 30 18:33:17.305640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.305607 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm" Oct 30 18:33:17.325376 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.325347 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1053887b-8b36-407c-8427-af7207e989b6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1053887b-8b36-407c-8427-af7207e989b6-plugins-config-volume\") pod \"1053887b-8b36-407c-8427-af7207e989b6\" (UID: \"1053887b-8b36-407c-8427-af7207e989b6\") " Oct 30 18:33:17.325581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.325441 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7703d89f-2407-4aa1-b263-1de901a8162b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7703d89f-2407-4aa1-b263-1de901a8162b-plugins-config-volume\") pod \"7703d89f-2407-4aa1-b263-1de901a8162b\" (UID: \"7703d89f-2407-4aa1-b263-1de901a8162b\") " Oct 30 18:33:17.325581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.325486 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1053887b-8b36-407c-8427-af7207e989b6-kube-api-access-c8tzn\" (UniqueName: \"kubernetes.io/projected/1053887b-8b36-407c-8427-af7207e989b6-kube-api-access-c8tzn\") pod \"1053887b-8b36-407c-8427-af7207e989b6\" (UID: \"1053887b-8b36-407c-8427-af7207e989b6\") " Oct 30 18:33:17.325581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.325554 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7703d89f-2407-4aa1-b263-1de901a8162b-kube-api-access-pt877\" (UniqueName: \"kubernetes.io/projected/7703d89f-2407-4aa1-b263-1de901a8162b-kube-api-access-pt877\") pod \"7703d89f-2407-4aa1-b263-1de901a8162b\" (UID: \"7703d89f-2407-4aa1-b263-1de901a8162b\") " Oct 30 18:33:17.325820 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.325789 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1053887b-8b36-407c-8427-af7207e989b6-plugins-config-volume" pod "1053887b-8b36-407c-8427-af7207e989b6" (UID: "1053887b-8b36-407c-8427-af7207e989b6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:17.325888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.325871 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7703d89f-2407-4aa1-b263-1de901a8162b-plugins-config-volume" pod "7703d89f-2407-4aa1-b263-1de901a8162b" (UID: "7703d89f-2407-4aa1-b263-1de901a8162b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:17.325985 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.325969 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1053887b-8b36-407c-8427-af7207e989b6-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:17.326037 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.325990 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7703d89f-2407-4aa1-b263-1de901a8162b-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:17.327892 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.327871 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7703d89f-2407-4aa1-b263-1de901a8162b-kube-api-access-pt877" pod "7703d89f-2407-4aa1-b263-1de901a8162b" (UID: "7703d89f-2407-4aa1-b263-1de901a8162b"). InnerVolumeSpecName "kube-api-access-pt877". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:17.328161 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.328145 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1053887b-8b36-407c-8427-af7207e989b6-kube-api-access-c8tzn" pod "1053887b-8b36-407c-8427-af7207e989b6" (UID: "1053887b-8b36-407c-8427-af7207e989b6"). InnerVolumeSpecName "kube-api-access-c8tzn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:17.400199 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.400162 5232 generic.go:356] "Generic (PLEG): container finished" podID="7703d89f-2407-4aa1-b263-1de901a8162b" containerID="eca962e46774c2c90c0be38cec888918908821719aae4bd861a922a10ae23ba5" exitCode=0 Oct 30 18:33:17.400372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.400213 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6" Oct 30 18:33:17.400372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.400247 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6" event={"ID":"7703d89f-2407-4aa1-b263-1de901a8162b","Type":"ContainerDied","Data":"eca962e46774c2c90c0be38cec888918908821719aae4bd861a922a10ae23ba5"} Oct 30 18:33:17.400372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.400301 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6" event={"ID":"7703d89f-2407-4aa1-b263-1de901a8162b","Type":"ContainerDied","Data":"623a8c0063d5dd9d06477417de26c911513acc1fd7f685fd232487628fe65d96"} Oct 30 18:33:17.400372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.400330 5232 scope.go:122] "RemoveContainer" containerID="eca962e46774c2c90c0be38cec888918908821719aae4bd861a922a10ae23ba5" Oct 30 18:33:17.401797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.401770 5232 generic.go:356] "Generic (PLEG): container finished" podID="1053887b-8b36-407c-8427-af7207e989b6" containerID="dcbf7007f6177639bc92b41a727528c37fa3db5e12d7bff1b7d07d55a874ffde" exitCode=0 Oct 30 18:33:17.401908 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.401813 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm" Oct 30 18:33:17.401908 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.401869 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm" event={"ID":"1053887b-8b36-407c-8427-af7207e989b6","Type":"ContainerDied","Data":"dcbf7007f6177639bc92b41a727528c37fa3db5e12d7bff1b7d07d55a874ffde"} Oct 30 18:33:17.402075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.401941 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm" event={"ID":"1053887b-8b36-407c-8427-af7207e989b6","Type":"ContainerDied","Data":"cb610cd309e4b1cf39164ea26aef0d03e5de0a083ce4d8262c5b29fd392f4c2c"} Oct 30 18:33:17.411405 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.411390 5232 scope.go:122] "RemoveContainer" containerID="eca962e46774c2c90c0be38cec888918908821719aae4bd861a922a10ae23ba5" Oct 30 18:33:17.411981 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:33:17.411953 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"eca962e46774c2c90c0be38cec888918908821719aae4bd861a922a10ae23ba5\": container with ID starting with eca962e46774c2c90c0be38cec888918908821719aae4bd861a922a10ae23ba5 not found: ID does not exist" containerID="eca962e46774c2c90c0be38cec888918908821719aae4bd861a922a10ae23ba5" Oct 30 18:33:17.412100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.411994 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"eca962e46774c2c90c0be38cec888918908821719aae4bd861a922a10ae23ba5"} err="failed to get container status \"eca962e46774c2c90c0be38cec888918908821719aae4bd861a922a10ae23ba5\": rpc error: code = NotFound desc = could not find container \"eca962e46774c2c90c0be38cec888918908821719aae4bd861a922a10ae23ba5\": container with ID starting with eca962e46774c2c90c0be38cec888918908821719aae4bd861a922a10ae23ba5 not found: ID does not exist" Oct 30 18:33:17.412100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.412018 5232 scope.go:122] "RemoveContainer" containerID="dcbf7007f6177639bc92b41a727528c37fa3db5e12d7bff1b7d07d55a874ffde" Oct 30 18:33:17.420924 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.420905 5232 scope.go:122] "RemoveContainer" containerID="dcbf7007f6177639bc92b41a727528c37fa3db5e12d7bff1b7d07d55a874ffde" Oct 30 18:33:17.421217 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:33:17.421194 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dcbf7007f6177639bc92b41a727528c37fa3db5e12d7bff1b7d07d55a874ffde\": container with ID starting with dcbf7007f6177639bc92b41a727528c37fa3db5e12d7bff1b7d07d55a874ffde not found: ID does not exist" containerID="dcbf7007f6177639bc92b41a727528c37fa3db5e12d7bff1b7d07d55a874ffde" Oct 30 18:33:17.421309 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.421223 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dcbf7007f6177639bc92b41a727528c37fa3db5e12d7bff1b7d07d55a874ffde"} err="failed to get container status \"dcbf7007f6177639bc92b41a727528c37fa3db5e12d7bff1b7d07d55a874ffde\": rpc error: code = NotFound desc = could not find container \"dcbf7007f6177639bc92b41a727528c37fa3db5e12d7bff1b7d07d55a874ffde\": container with ID starting with dcbf7007f6177639bc92b41a727528c37fa3db5e12d7bff1b7d07d55a874ffde not found: ID does not exist" Oct 30 18:33:17.426969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.426944 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8tzn\" (UniqueName: \"kubernetes.io/projected/1053887b-8b36-407c-8427-af7207e989b6-kube-api-access-c8tzn\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:17.426969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.426967 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pt877\" (UniqueName: \"kubernetes.io/projected/7703d89f-2407-4aa1-b263-1de901a8162b-kube-api-access-pt877\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:17.451790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.451762 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6"] Oct 30 18:33:17.453998 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.453978 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-c22995-74cf4888c6-kp7h6"] Oct 30 18:33:17.487263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.487241 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm"] Oct 30 18:33:17.492897 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:17.492873 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-c22995-7896c5b7d7-w9dgm"] Oct 30 18:33:18.339412 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:18.339003 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88"] Oct 30 18:33:18.339412 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:18.339237 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" podUID="b187a263-e999-46e7-a1d6-4a44595f6e2d" containerName="maas-api" containerID="cri-o://5cd3b035bd947ccec0c63d5f43bb7a88151f964760a8175d90adec1f1c63828b" gracePeriod=30 Oct 30 18:33:18.629593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:18.629569 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" Oct 30 18:33:18.638768 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:18.638744 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b187a263-e999-46e7-a1d6-4a44595f6e2d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b187a263-e999-46e7-a1d6-4a44595f6e2d-openshift-service-ca\") pod \"b187a263-e999-46e7-a1d6-4a44595f6e2d\" (UID: \"b187a263-e999-46e7-a1d6-4a44595f6e2d\") " Oct 30 18:33:18.638903 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:18.638832 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b187a263-e999-46e7-a1d6-4a44595f6e2d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b187a263-e999-46e7-a1d6-4a44595f6e2d-maas-api-tls\") pod \"b187a263-e999-46e7-a1d6-4a44595f6e2d\" (UID: \"b187a263-e999-46e7-a1d6-4a44595f6e2d\") " Oct 30 18:33:18.638965 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:18.638943 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b187a263-e999-46e7-a1d6-4a44595f6e2d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/b187a263-e999-46e7-a1d6-4a44595f6e2d-metrics-certs\") pod \"b187a263-e999-46e7-a1d6-4a44595f6e2d\" (UID: \"b187a263-e999-46e7-a1d6-4a44595f6e2d\") " Oct 30 18:33:18.639036 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:18.639004 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b187a263-e999-46e7-a1d6-4a44595f6e2d-kube-api-access-t5rb8\" (UniqueName: \"kubernetes.io/projected/b187a263-e999-46e7-a1d6-4a44595f6e2d-kube-api-access-t5rb8\") pod \"b187a263-e999-46e7-a1d6-4a44595f6e2d\" (UID: \"b187a263-e999-46e7-a1d6-4a44595f6e2d\") " Oct 30 18:33:18.639096 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:18.639050 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b187a263-e999-46e7-a1d6-4a44595f6e2d-openshift-service-ca" pod "b187a263-e999-46e7-a1d6-4a44595f6e2d" (UID: "b187a263-e999-46e7-a1d6-4a44595f6e2d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:18.639316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:18.639300 5232 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/b187a263-e999-46e7-a1d6-4a44595f6e2d-openshift-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:18.641017 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:18.640993 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b187a263-e999-46e7-a1d6-4a44595f6e2d-maas-api-tls" pod "b187a263-e999-46e7-a1d6-4a44595f6e2d" (UID: "b187a263-e999-46e7-a1d6-4a44595f6e2d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:18.641148 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:18.641129 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b187a263-e999-46e7-a1d6-4a44595f6e2d-kube-api-access-t5rb8" pod "b187a263-e999-46e7-a1d6-4a44595f6e2d" (UID: "b187a263-e999-46e7-a1d6-4a44595f6e2d"). InnerVolumeSpecName "kube-api-access-t5rb8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:18.641214 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:18.641158 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b187a263-e999-46e7-a1d6-4a44595f6e2d-metrics-certs" pod "b187a263-e999-46e7-a1d6-4a44595f6e2d" (UID: "b187a263-e999-46e7-a1d6-4a44595f6e2d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:18.740943 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:18.740898 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t5rb8\" (UniqueName: \"kubernetes.io/projected/b187a263-e999-46e7-a1d6-4a44595f6e2d-kube-api-access-t5rb8\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:18.740943 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:18.740939 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/b187a263-e999-46e7-a1d6-4a44595f6e2d-maas-api-tls\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:18.740943 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:18.740952 5232 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/b187a263-e999-46e7-a1d6-4a44595f6e2d-metrics-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:18.841183 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:18.841025 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1053887b-8b36-407c-8427-af7207e989b6" path="/var/lib/kubelet/pods/1053887b-8b36-407c-8427-af7207e989b6/volumes" Oct 30 18:33:18.841669 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:18.841639 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7703d89f-2407-4aa1-b263-1de901a8162b" path="/var/lib/kubelet/pods/7703d89f-2407-4aa1-b263-1de901a8162b/volumes" Oct 30 18:33:19.013818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:19.013772 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg"] Oct 30 18:33:19.016048 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:19.016022 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-c22995-krztg"] Oct 30 18:33:19.417748 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:19.417658 5232 generic.go:356] "Generic (PLEG): container finished" podID="b187a263-e999-46e7-a1d6-4a44595f6e2d" containerID="5cd3b035bd947ccec0c63d5f43bb7a88151f964760a8175d90adec1f1c63828b" exitCode=0 Oct 30 18:33:19.417748 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:19.417731 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" Oct 30 18:33:19.418295 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:19.417749 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" event={"ID":"b187a263-e999-46e7-a1d6-4a44595f6e2d","Type":"ContainerDied","Data":"5cd3b035bd947ccec0c63d5f43bb7a88151f964760a8175d90adec1f1c63828b"} Oct 30 18:33:19.418295 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:19.417815 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88" event={"ID":"b187a263-e999-46e7-a1d6-4a44595f6e2d","Type":"ContainerDied","Data":"bcaee940241f66a64e8622fa24c5b5f3698212c8e855748ecbdcfb0d46e6b8ec"} Oct 30 18:33:19.418295 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:19.417849 5232 scope.go:122] "RemoveContainer" containerID="5cd3b035bd947ccec0c63d5f43bb7a88151f964760a8175d90adec1f1c63828b" Oct 30 18:33:19.427809 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:19.427776 5232 scope.go:122] "RemoveContainer" containerID="5cd3b035bd947ccec0c63d5f43bb7a88151f964760a8175d90adec1f1c63828b" Oct 30 18:33:19.428090 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:33:19.428066 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5cd3b035bd947ccec0c63d5f43bb7a88151f964760a8175d90adec1f1c63828b\": container with ID starting with 5cd3b035bd947ccec0c63d5f43bb7a88151f964760a8175d90adec1f1c63828b not found: ID does not exist" containerID="5cd3b035bd947ccec0c63d5f43bb7a88151f964760a8175d90adec1f1c63828b" Oct 30 18:33:19.428137 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:19.428101 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5cd3b035bd947ccec0c63d5f43bb7a88151f964760a8175d90adec1f1c63828b"} err="failed to get container status \"5cd3b035bd947ccec0c63d5f43bb7a88151f964760a8175d90adec1f1c63828b\": rpc error: code = NotFound desc = could not find container \"5cd3b035bd947ccec0c63d5f43bb7a88151f964760a8175d90adec1f1c63828b\": container with ID starting with 5cd3b035bd947ccec0c63d5f43bb7a88151f964760a8175d90adec1f1c63828b not found: ID does not exist" Oct 30 18:33:19.468052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:19.468020 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88"] Oct 30 18:33:19.479921 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:19.479883 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w4-c22995-6cc8c86f94-7mv88"] Oct 30 18:33:20.801656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:20.801584 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="45e57987-d470-42b7-8547-be295cc129e5" path="/var/lib/kubelet/pods/45e57987-d470-42b7-8547-be295cc129e5/volumes" Oct 30 18:33:20.802040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:20.802014 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b187a263-e999-46e7-a1d6-4a44595f6e2d" path="/var/lib/kubelet/pods/b187a263-e999-46e7-a1d6-4a44595f6e2d/volumes" Oct 30 18:33:21.910892 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.910861 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7"] Oct 30 18:33:21.911480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.911459 5232 cpu_manager.go:414] "containerMap: removing container" podUID="061ebd64-fd21-4ff1-ae1e-27148f48aa3c" containerName="payload-pre-processing" Oct 30 18:33:21.911588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.911483 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="061ebd64-fd21-4ff1-ae1e-27148f48aa3c" containerName="payload-pre-processing" Oct 30 18:33:21.911588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.911554 5232 cpu_manager.go:414] "containerMap: removing container" podUID="b187a263-e999-46e7-a1d6-4a44595f6e2d" containerName="maas-api" Oct 30 18:33:21.911588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.911562 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b187a263-e999-46e7-a1d6-4a44595f6e2d" containerName="maas-api" Oct 30 18:33:21.911588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.911575 5232 cpu_manager.go:414] "containerMap: removing container" podUID="cf57d029-4ffe-44e4-b025-f28c296405b2" containerName="revoke-keys" Oct 30 18:33:21.911588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.911584 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf57d029-4ffe-44e4-b025-f28c296405b2" containerName="revoke-keys" Oct 30 18:33:21.911869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.911603 5232 cpu_manager.go:414] "containerMap: removing container" podUID="1053887b-8b36-407c-8427-af7207e989b6" containerName="payload-pre-processing" Oct 30 18:33:21.911869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.911611 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1053887b-8b36-407c-8427-af7207e989b6" containerName="payload-pre-processing" Oct 30 18:33:21.911869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.911639 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7703d89f-2407-4aa1-b263-1de901a8162b" containerName="payload-processing" Oct 30 18:33:21.911869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.911648 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7703d89f-2407-4aa1-b263-1de901a8162b" containerName="payload-processing" Oct 30 18:33:21.911869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.911763 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="b187a263-e999-46e7-a1d6-4a44595f6e2d" containerName="maas-api" Oct 30 18:33:21.911869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.911777 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf57d029-4ffe-44e4-b025-f28c296405b2" containerName="revoke-keys" Oct 30 18:33:21.911869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.911789 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="1053887b-8b36-407c-8427-af7207e989b6" containerName="payload-pre-processing" Oct 30 18:33:21.911869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.911800 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7703d89f-2407-4aa1-b263-1de901a8162b" containerName="payload-processing" Oct 30 18:33:21.911869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.911811 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="061ebd64-fd21-4ff1-ae1e-27148f48aa3c" containerName="payload-pre-processing" Oct 30 18:33:21.916865 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.916838 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7" Oct 30 18:33:21.935415 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.935381 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:33:21.935415 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.935410 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:33:21.941393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.941343 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" podUID="048571d7-450a-40b4-8c18-94ec10221b86" containerName="payload-processing" containerID="cri-o://2321ac6555ed9e7b126d2a02fc840331f148df60159a950adb5261d090c50ca8" gracePeriod=25 Oct 30 18:33:21.951236 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.951209 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5"] Oct 30 18:33:21.954981 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.954965 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7"] Oct 30 18:33:21.955090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.955053 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5" Oct 30 18:33:21.963462 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.963430 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5"] Oct 30 18:33:21.974553 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.974496 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" podUID="1f3f54d2-9382-4abd-81a5-4fba85c28dad" containerName="payload-pre-processing" containerID="cri-o://b6f8d0ddbb6dad0de774503616b83ab4ec48d5c7c6d3c37dcf9a0d34f7558eea" gracePeriod=25 Oct 30 18:33:21.975432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.975405 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wf57r\" (UniqueName: \"kubernetes.io/projected/317b89c4-eac3-48eb-8587-a3911c5dd435-kube-api-access-wf57r\") pod \"maas-api-revoke-sub-7286092f-hs2q7\" (UID: \"317b89c4-eac3-48eb-8587-a3911c5dd435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7" Oct 30 18:33:21.975579 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.975449 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d46be40b-6bc7-4633-8389-f7a4d79c603d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e816d55-6sxz5\" (UID: \"d46be40b-6bc7-4633-8389-f7a4d79c603d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5" Oct 30 18:33:21.975579 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.975561 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317b89c4-eac3-48eb-8587-a3911c5dd435-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7286092f-hs2q7\" (UID: \"317b89c4-eac3-48eb-8587-a3911c5dd435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7" Oct 30 18:33:21.975731 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.975711 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcv96\" (UniqueName: \"kubernetes.io/projected/d46be40b-6bc7-4633-8389-f7a4d79c603d-kube-api-access-mcv96\") pod \"maas-api-revoke-sub-6e816d55-6sxz5\" (UID: \"d46be40b-6bc7-4633-8389-f7a4d79c603d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5" Oct 30 18:33:21.975800 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.975753 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317b89c4-eac3-48eb-8587-a3911c5dd435-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7286092f-hs2q7\" (UID: \"317b89c4-eac3-48eb-8587-a3911c5dd435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7" Oct 30 18:33:21.975800 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:21.975780 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d46be40b-6bc7-4633-8389-f7a4d79c603d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e816d55-6sxz5\" (UID: \"d46be40b-6bc7-4633-8389-f7a4d79c603d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5" Oct 30 18:33:22.077145 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:22.077106 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317b89c4-eac3-48eb-8587-a3911c5dd435-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7286092f-hs2q7\" (UID: \"317b89c4-eac3-48eb-8587-a3911c5dd435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7" Oct 30 18:33:22.077145 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:22.077147 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d46be40b-6bc7-4633-8389-f7a4d79c603d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e816d55-6sxz5\" (UID: \"d46be40b-6bc7-4633-8389-f7a4d79c603d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5" Oct 30 18:33:22.077399 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:22.077253 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wf57r\" (UniqueName: \"kubernetes.io/projected/317b89c4-eac3-48eb-8587-a3911c5dd435-kube-api-access-wf57r\") pod \"maas-api-revoke-sub-7286092f-hs2q7\" (UID: \"317b89c4-eac3-48eb-8587-a3911c5dd435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7" Oct 30 18:33:22.077399 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:22.077279 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d46be40b-6bc7-4633-8389-f7a4d79c603d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e816d55-6sxz5\" (UID: \"d46be40b-6bc7-4633-8389-f7a4d79c603d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5" Oct 30 18:33:22.077399 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:22.077307 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317b89c4-eac3-48eb-8587-a3911c5dd435-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7286092f-hs2q7\" (UID: \"317b89c4-eac3-48eb-8587-a3911c5dd435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7" Oct 30 18:33:22.077621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:22.077547 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcv96\" (UniqueName: \"kubernetes.io/projected/d46be40b-6bc7-4633-8389-f7a4d79c603d-kube-api-access-mcv96\") pod \"maas-api-revoke-sub-6e816d55-6sxz5\" (UID: \"d46be40b-6bc7-4633-8389-f7a4d79c603d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5" Oct 30 18:33:22.078067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:22.078042 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317b89c4-eac3-48eb-8587-a3911c5dd435-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-7286092f-hs2q7\" (UID: \"317b89c4-eac3-48eb-8587-a3911c5dd435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7" Oct 30 18:33:22.078284 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:22.078260 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d46be40b-6bc7-4633-8389-f7a4d79c603d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6e816d55-6sxz5\" (UID: \"d46be40b-6bc7-4633-8389-f7a4d79c603d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5" Oct 30 18:33:22.079769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:22.079748 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d46be40b-6bc7-4633-8389-f7a4d79c603d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6e816d55-6sxz5\" (UID: \"d46be40b-6bc7-4633-8389-f7a4d79c603d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5" Oct 30 18:33:22.079846 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:22.079795 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317b89c4-eac3-48eb-8587-a3911c5dd435-maas-api-xks-ca\") pod \"maas-api-revoke-sub-7286092f-hs2q7\" (UID: \"317b89c4-eac3-48eb-8587-a3911c5dd435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7" Oct 30 18:33:22.096305 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:22.096276 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcv96\" (UniqueName: \"kubernetes.io/projected/d46be40b-6bc7-4633-8389-f7a4d79c603d-kube-api-access-mcv96\") pod \"maas-api-revoke-sub-6e816d55-6sxz5\" (UID: \"d46be40b-6bc7-4633-8389-f7a4d79c603d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5" Oct 30 18:33:22.097039 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:22.097021 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wf57r\" (UniqueName: \"kubernetes.io/projected/317b89c4-eac3-48eb-8587-a3911c5dd435-kube-api-access-wf57r\") pod \"maas-api-revoke-sub-7286092f-hs2q7\" (UID: \"317b89c4-eac3-48eb-8587-a3911c5dd435\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7" Oct 30 18:33:22.230243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:22.230175 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7" Oct 30 18:33:22.279028 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:22.278987 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5" Oct 30 18:33:22.525485 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:22.525450 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7"] Oct 30 18:33:22.623612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:22.623585 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5"] Oct 30 18:33:22.625029 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:33:22.624999 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd46be40b_6bc7_4633_8389_f7a4d79c603d.slice/crio-2954fb4cbbd6695935105233d4f041a57119cd648e84f75f7c44d1e205bdef96 WatchSource:0}: Error finding container 2954fb4cbbd6695935105233d4f041a57119cd648e84f75f7c44d1e205bdef96: Status 404 returned error can't find the container with id 2954fb4cbbd6695935105233d4f041a57119cd648e84f75f7c44d1e205bdef96 Oct 30 18:33:23.446598 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:23.446555 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7" event={"ID":"317b89c4-eac3-48eb-8587-a3911c5dd435","Type":"ContainerStarted","Data":"993e8e5bfb6e39aeeba2291eae428d565579c3c63d15b154d9b8dc1911533d31"} Oct 30 18:33:23.447040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:23.446676 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7" event={"ID":"317b89c4-eac3-48eb-8587-a3911c5dd435","Type":"ContainerStarted","Data":"082f984b43eb32990523dbf210bee7b69809186ce9581da697e23d7108a9acdd"} Oct 30 18:33:23.448565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:23.448536 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5" event={"ID":"d46be40b-6bc7-4633-8389-f7a4d79c603d","Type":"ContainerStarted","Data":"64eea3b4936d5047087fac8a2b3af95e815ca831afd7136c1e894d077a2bb943"} Oct 30 18:33:23.448675 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:23.448590 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5" event={"ID":"d46be40b-6bc7-4633-8389-f7a4d79c603d","Type":"ContainerStarted","Data":"2954fb4cbbd6695935105233d4f041a57119cd648e84f75f7c44d1e205bdef96"} Oct 30 18:33:23.488588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:23.488534 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7" podStartSLOduration=2.488501163 podStartE2EDuration="2.488501163s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:23.486005883 +0000 UTC m=+1902.802426216" watchObservedRunningTime="2026-10-30 18:33:23.488501163 +0000 UTC m=+1902.804921505" Oct 30 18:33:23.524411 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:23.524356 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5" podStartSLOduration=2.524342281 podStartE2EDuration="2.524342281s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:23.523714841 +0000 UTC m=+1902.840135164" watchObservedRunningTime="2026-10-30 18:33:23.524342281 +0000 UTC m=+1902.840762601" Oct 30 18:33:24.455604 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:24.455566 5232 generic.go:356] "Generic (PLEG): container finished" podID="317b89c4-eac3-48eb-8587-a3911c5dd435" containerID="993e8e5bfb6e39aeeba2291eae428d565579c3c63d15b154d9b8dc1911533d31" exitCode=0 Oct 30 18:33:24.456155 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:24.455659 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7" event={"ID":"317b89c4-eac3-48eb-8587-a3911c5dd435","Type":"ContainerDied","Data":"993e8e5bfb6e39aeeba2291eae428d565579c3c63d15b154d9b8dc1911533d31"} Oct 30 18:33:24.457299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:24.457274 5232 generic.go:356] "Generic (PLEG): container finished" podID="d46be40b-6bc7-4633-8389-f7a4d79c603d" containerID="64eea3b4936d5047087fac8a2b3af95e815ca831afd7136c1e894d077a2bb943" exitCode=0 Oct 30 18:33:24.457417 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:24.457353 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5" event={"ID":"d46be40b-6bc7-4633-8389-f7a4d79c603d","Type":"ContainerDied","Data":"64eea3b4936d5047087fac8a2b3af95e815ca831afd7136c1e894d077a2bb943"} Oct 30 18:33:25.666002 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.665974 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5" Oct 30 18:33:25.669585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.669566 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7" Oct 30 18:33:25.731684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.731593 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d46be40b-6bc7-4633-8389-f7a4d79c603d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d46be40b-6bc7-4633-8389-f7a4d79c603d-maas-api-xks-ca\") pod \"d46be40b-6bc7-4633-8389-f7a4d79c603d\" (UID: \"d46be40b-6bc7-4633-8389-f7a4d79c603d\") " Oct 30 18:33:25.731684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.731650 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d46be40b-6bc7-4633-8389-f7a4d79c603d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d46be40b-6bc7-4633-8389-f7a4d79c603d-maas-api-openshift-ca\") pod \"d46be40b-6bc7-4633-8389-f7a4d79c603d\" (UID: \"d46be40b-6bc7-4633-8389-f7a4d79c603d\") " Oct 30 18:33:25.731684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.731682 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/317b89c4-eac3-48eb-8587-a3911c5dd435-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317b89c4-eac3-48eb-8587-a3911c5dd435-maas-api-openshift-ca\") pod \"317b89c4-eac3-48eb-8587-a3911c5dd435\" (UID: \"317b89c4-eac3-48eb-8587-a3911c5dd435\") " Oct 30 18:33:25.731976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.731807 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d46be40b-6bc7-4633-8389-f7a4d79c603d-kube-api-access-mcv96\" (UniqueName: \"kubernetes.io/projected/d46be40b-6bc7-4633-8389-f7a4d79c603d-kube-api-access-mcv96\") pod \"d46be40b-6bc7-4633-8389-f7a4d79c603d\" (UID: \"d46be40b-6bc7-4633-8389-f7a4d79c603d\") " Oct 30 18:33:25.731976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.731849 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/317b89c4-eac3-48eb-8587-a3911c5dd435-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317b89c4-eac3-48eb-8587-a3911c5dd435-maas-api-xks-ca\") pod \"317b89c4-eac3-48eb-8587-a3911c5dd435\" (UID: \"317b89c4-eac3-48eb-8587-a3911c5dd435\") " Oct 30 18:33:25.731976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.731892 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/317b89c4-eac3-48eb-8587-a3911c5dd435-kube-api-access-wf57r\" (UniqueName: \"kubernetes.io/projected/317b89c4-eac3-48eb-8587-a3911c5dd435-kube-api-access-wf57r\") pod \"317b89c4-eac3-48eb-8587-a3911c5dd435\" (UID: \"317b89c4-eac3-48eb-8587-a3911c5dd435\") " Oct 30 18:33:25.732167 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.732071 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d46be40b-6bc7-4633-8389-f7a4d79c603d-maas-api-openshift-ca" pod "d46be40b-6bc7-4633-8389-f7a4d79c603d" (UID: "d46be40b-6bc7-4633-8389-f7a4d79c603d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:25.732167 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.732152 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/317b89c4-eac3-48eb-8587-a3911c5dd435-maas-api-openshift-ca" pod "317b89c4-eac3-48eb-8587-a3911c5dd435" (UID: "317b89c4-eac3-48eb-8587-a3911c5dd435"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:25.732356 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.732336 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d46be40b-6bc7-4633-8389-f7a4d79c603d-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:25.732411 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.732364 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/317b89c4-eac3-48eb-8587-a3911c5dd435-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:25.734156 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.734120 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/317b89c4-eac3-48eb-8587-a3911c5dd435-maas-api-xks-ca" pod "317b89c4-eac3-48eb-8587-a3911c5dd435" (UID: "317b89c4-eac3-48eb-8587-a3911c5dd435"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:25.734690 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.734658 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d46be40b-6bc7-4633-8389-f7a4d79c603d-maas-api-xks-ca" pod "d46be40b-6bc7-4633-8389-f7a4d79c603d" (UID: "d46be40b-6bc7-4633-8389-f7a4d79c603d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:25.734774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.734745 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/317b89c4-eac3-48eb-8587-a3911c5dd435-kube-api-access-wf57r" pod "317b89c4-eac3-48eb-8587-a3911c5dd435" (UID: "317b89c4-eac3-48eb-8587-a3911c5dd435"). InnerVolumeSpecName "kube-api-access-wf57r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:25.734814 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.734776 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d46be40b-6bc7-4633-8389-f7a4d79c603d-kube-api-access-mcv96" pod "d46be40b-6bc7-4633-8389-f7a4d79c603d" (UID: "d46be40b-6bc7-4633-8389-f7a4d79c603d"). InnerVolumeSpecName "kube-api-access-mcv96". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:25.833730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.833694 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mcv96\" (UniqueName: \"kubernetes.io/projected/d46be40b-6bc7-4633-8389-f7a4d79c603d-kube-api-access-mcv96\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:25.833730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.833724 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/317b89c4-eac3-48eb-8587-a3911c5dd435-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:25.833730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.833733 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wf57r\" (UniqueName: \"kubernetes.io/projected/317b89c4-eac3-48eb-8587-a3911c5dd435-kube-api-access-wf57r\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:25.833981 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:25.833742 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d46be40b-6bc7-4633-8389-f7a4d79c603d-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.261195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.253130 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" containerID="cri-o://e96ffc5a00b0f002bc7762f5d7d7b1b1636ce2855821df81940ea3ba81ecb9a2" gracePeriod=45 Oct 30 18:33:26.275990 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.275951 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" containerID="cri-o://0baea61d2cdf1633f7fcc5db6c53bd3b70cf82be5427f4ceab0063e53d26b468" gracePeriod=45 Oct 30 18:33:26.486900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.485805 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7" Oct 30 18:33:26.486900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.485835 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7" event={"ID":"317b89c4-eac3-48eb-8587-a3911c5dd435","Type":"ContainerDied","Data":"082f984b43eb32990523dbf210bee7b69809186ce9581da697e23d7108a9acdd"} Oct 30 18:33:26.486900 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.485884 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="082f984b43eb32990523dbf210bee7b69809186ce9581da697e23d7108a9acdd" Oct 30 18:33:26.489280 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.489003 5232 generic.go:356] "Generic (PLEG): container finished" podID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerID="0baea61d2cdf1633f7fcc5db6c53bd3b70cf82be5427f4ceab0063e53d26b468" exitCode=0 Oct 30 18:33:26.489280 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.489200 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" event={"ID":"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b","Type":"ContainerDied","Data":"0baea61d2cdf1633f7fcc5db6c53bd3b70cf82be5427f4ceab0063e53d26b468"} Oct 30 18:33:26.492558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.491970 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2"] Oct 30 18:33:26.492558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.492176 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" podUID="7b16e19a-e62c-47b4-b695-ccc7164524b5" containerName="istio-proxy" containerID="cri-o://887adb3ce5d62083c4399327087dabba8ad5131dfda0bd3082011f56093cc782" gracePeriod=30 Oct 30 18:33:26.493214 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.493190 5232 generic.go:356] "Generic (PLEG): container finished" podID="73249ea3-3ad3-489d-ac99-b1145098e719" containerID="e96ffc5a00b0f002bc7762f5d7d7b1b1636ce2855821df81940ea3ba81ecb9a2" exitCode=0 Oct 30 18:33:26.493318 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.493285 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" event={"ID":"73249ea3-3ad3-489d-ac99-b1145098e719","Type":"ContainerDied","Data":"e96ffc5a00b0f002bc7762f5d7d7b1b1636ce2855821df81940ea3ba81ecb9a2"} Oct 30 18:33:26.495672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.495652 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5" Oct 30 18:33:26.495852 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.495824 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5" event={"ID":"d46be40b-6bc7-4633-8389-f7a4d79c603d","Type":"ContainerDied","Data":"2954fb4cbbd6695935105233d4f041a57119cd648e84f75f7c44d1e205bdef96"} Oct 30 18:33:26.495983 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.495867 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2954fb4cbbd6695935105233d4f041a57119cd648e84f75f7c44d1e205bdef96" Oct 30 18:33:26.549088 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.549055 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:33:26.576241 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.576211 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:33:26.649422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.649373 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-dshm\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-dshm\") pod \"73249ea3-3ad3-489d-ac99-b1145098e719\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " Oct 30 18:33:26.649628 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.649437 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-model-cache\") pod \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " Oct 30 18:33:26.649628 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.649482 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-tls-certs\") pod \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " Oct 30 18:33:26.649628 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.649578 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-home\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-home\") pod \"73249ea3-3ad3-489d-ac99-b1145098e719\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " Oct 30 18:33:26.649628 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.649608 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-tmp-dir\") pod \"73249ea3-3ad3-489d-ac99-b1145098e719\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " Oct 30 18:33:26.649799 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.649645 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-dshm\") pod \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " Oct 30 18:33:26.649799 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.649679 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73249ea3-3ad3-489d-ac99-b1145098e719-tls-certs\" (UniqueName: \"kubernetes.io/secret/73249ea3-3ad3-489d-ac99-b1145098e719-tls-certs\") pod \"73249ea3-3ad3-489d-ac99-b1145098e719\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " Oct 30 18:33:26.649799 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.649732 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-home\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-home\") pod \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " Oct 30 18:33:26.649799 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.649761 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-model-cache" pod "b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" (UID: "b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:26.649799 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.649777 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-tmp-dir\") pod \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " Oct 30 18:33:26.649988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.649828 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-model-cache\") pod \"73249ea3-3ad3-489d-ac99-b1145098e719\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " Oct 30 18:33:26.649988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.649856 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73249ea3-3ad3-489d-ac99-b1145098e719-kube-api-access-tbdk5\" (UniqueName: \"kubernetes.io/projected/73249ea3-3ad3-489d-ac99-b1145098e719-kube-api-access-tbdk5\") pod \"73249ea3-3ad3-489d-ac99-b1145098e719\" (UID: \"73249ea3-3ad3-489d-ac99-b1145098e719\") " Oct 30 18:33:26.649988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.649944 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-kube-api-access-hzvvt\" (UniqueName: \"kubernetes.io/projected/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-kube-api-access-hzvvt\") pod \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\" (UID: \"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b\") " Oct 30 18:33:26.649988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.649947 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-tmp-dir" pod "73249ea3-3ad3-489d-ac99-b1145098e719" (UID: "73249ea3-3ad3-489d-ac99-b1145098e719"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:26.650368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.650331 5232 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-model-cache\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.650368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.650356 5232 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-tmp-dir\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.650816 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.650560 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-home" pod "b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" (UID: "b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:26.651410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.651106 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-home" pod "73249ea3-3ad3-489d-ac99-b1145098e719" (UID: "73249ea3-3ad3-489d-ac99-b1145098e719"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:26.651410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.651216 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-tmp-dir" pod "b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" (UID: "b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:26.651615 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.651409 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-model-cache" pod "73249ea3-3ad3-489d-ac99-b1145098e719" (UID: "73249ea3-3ad3-489d-ac99-b1145098e719"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:26.657647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.655917 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-kube-api-access-hzvvt" pod "b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" (UID: "b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b"). InnerVolumeSpecName "kube-api-access-hzvvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:26.657647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.656041 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-tls-certs" pod "b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" (UID: "b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:26.660337 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.660022 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-dshm" pod "73249ea3-3ad3-489d-ac99-b1145098e719" (UID: "73249ea3-3ad3-489d-ac99-b1145098e719"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:26.673993 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.664878 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-dshm" pod "b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" (UID: "b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:26.673993 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.665129 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73249ea3-3ad3-489d-ac99-b1145098e719-tls-certs" pod "73249ea3-3ad3-489d-ac99-b1145098e719" (UID: "73249ea3-3ad3-489d-ac99-b1145098e719"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:26.677820 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.677685 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73249ea3-3ad3-489d-ac99-b1145098e719-kube-api-access-tbdk5" pod "73249ea3-3ad3-489d-ac99-b1145098e719" (UID: "73249ea3-3ad3-489d-ac99-b1145098e719"). InnerVolumeSpecName "kube-api-access-tbdk5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:26.751432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.751399 5232 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-home\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.751432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.751430 5232 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-tmp-dir\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.751432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.751439 5232 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-model-cache\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.751701 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.751449 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tbdk5\" (UniqueName: \"kubernetes.io/projected/73249ea3-3ad3-489d-ac99-b1145098e719-kube-api-access-tbdk5\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.751701 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.751458 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzvvt\" (UniqueName: \"kubernetes.io/projected/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-kube-api-access-hzvvt\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.751701 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.751467 5232 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-dshm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.751701 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.751474 5232 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-tls-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.751701 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.751482 5232 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/73249ea3-3ad3-489d-ac99-b1145098e719-home\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.751701 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.751488 5232 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b-dshm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:26.751701 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:26.751496 5232 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/73249ea3-3ad3-489d-ac99-b1145098e719-tls-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:27.034729 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:33:27.034692 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:33:27.099219 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.099137 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb"] Oct 30 18:33:27.099614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.099598 5232 cpu_manager.go:414] "containerMap: removing container" podUID="d46be40b-6bc7-4633-8389-f7a4d79c603d" containerName="revoke-keys" Oct 30 18:33:27.099693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.099615 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d46be40b-6bc7-4633-8389-f7a4d79c603d" containerName="revoke-keys" Oct 30 18:33:27.099693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.099647 5232 cpu_manager.go:414] "containerMap: removing container" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" Oct 30 18:33:27.099693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.099653 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" Oct 30 18:33:27.099693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.099665 5232 cpu_manager.go:414] "containerMap: removing container" podUID="317b89c4-eac3-48eb-8587-a3911c5dd435" containerName="revoke-keys" Oct 30 18:33:27.099693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.099670 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="317b89c4-eac3-48eb-8587-a3911c5dd435" containerName="revoke-keys" Oct 30 18:33:27.099693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.099678 5232 cpu_manager.go:414] "containerMap: removing container" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" Oct 30 18:33:27.099693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.099683 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" Oct 30 18:33:27.099922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.099748 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" containerName="main" Oct 30 18:33:27.099922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.099756 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="317b89c4-eac3-48eb-8587-a3911c5dd435" containerName="revoke-keys" Oct 30 18:33:27.099922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.099762 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" containerName="main" Oct 30 18:33:27.099922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.099770 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="d46be40b-6bc7-4633-8389-f7a4d79c603d" containerName="revoke-keys" Oct 30 18:33:27.104379 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.104360 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" Oct 30 18:33:27.119619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.119588 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:33:27.119780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.119616 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:33:27.121289 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.121257 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb"] Oct 30 18:33:27.155215 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.155191 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9zrds\" (UniqueName: \"kubernetes.io/projected/eecf0c23-83bb-42aa-bd59-18d9dae718e2-kube-api-access-9zrds\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb\" (UID: \"eecf0c23-83bb-42aa-bd59-18d9dae718e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" Oct 30 18:33:27.155370 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.155267 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecf0c23-83bb-42aa-bd59-18d9dae718e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb\" (UID: \"eecf0c23-83bb-42aa-bd59-18d9dae718e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" Oct 30 18:33:27.155420 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.155365 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecf0c23-83bb-42aa-bd59-18d9dae718e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb\" (UID: \"eecf0c23-83bb-42aa-bd59-18d9dae718e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" Oct 30 18:33:27.256212 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.256174 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecf0c23-83bb-42aa-bd59-18d9dae718e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb\" (UID: \"eecf0c23-83bb-42aa-bd59-18d9dae718e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" Oct 30 18:33:27.256402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.256231 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecf0c23-83bb-42aa-bd59-18d9dae718e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb\" (UID: \"eecf0c23-83bb-42aa-bd59-18d9dae718e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" Oct 30 18:33:27.256402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.256268 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9zrds\" (UniqueName: \"kubernetes.io/projected/eecf0c23-83bb-42aa-bd59-18d9dae718e2-kube-api-access-9zrds\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb\" (UID: \"eecf0c23-83bb-42aa-bd59-18d9dae718e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" Oct 30 18:33:27.257003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.256979 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecf0c23-83bb-42aa-bd59-18d9dae718e2-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb\" (UID: \"eecf0c23-83bb-42aa-bd59-18d9dae718e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" Oct 30 18:33:27.258749 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.258727 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecf0c23-83bb-42aa-bd59-18d9dae718e2-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb\" (UID: \"eecf0c23-83bb-42aa-bd59-18d9dae718e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" Oct 30 18:33:27.275617 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.275588 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9zrds\" (UniqueName: \"kubernetes.io/projected/eecf0c23-83bb-42aa-bd59-18d9dae718e2-kube-api-access-9zrds\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb\" (UID: \"eecf0c23-83bb-42aa-bd59-18d9dae718e2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" Oct 30 18:33:27.419838 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.419757 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" Oct 30 18:33:27.508303 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.508277 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" Oct 30 18:33:27.508303 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.508291 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6" event={"ID":"b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b","Type":"ContainerDied","Data":"e401daa9ba21039600043953c9fbd9135354325c7a45b635ca16159599f630a5"} Oct 30 18:33:27.508557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.508349 5232 scope.go:122] "RemoveContainer" containerID="0baea61d2cdf1633f7fcc5db6c53bd3b70cf82be5427f4ceab0063e53d26b468" Oct 30 18:33:27.510896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.510873 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" event={"ID":"73249ea3-3ad3-489d-ac99-b1145098e719","Type":"ContainerDied","Data":"65b73b1158764d0672754cc3f8ed8ee5750f7328b226d9cd700a9a669eabe7c3"} Oct 30 18:33:27.511038 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.510974 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2" Oct 30 18:33:27.520289 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.520273 5232 scope.go:122] "RemoveContainer" containerID="e96ffc5a00b0f002bc7762f5d7d7b1b1636ce2855821df81940ea3ba81ecb9a2" Oct 30 18:33:27.561627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.561598 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2"] Oct 30 18:33:27.564789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.564768 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/distinct-2-w2-fde2c5-kserve-64b79db897-szqf2"] Oct 30 18:33:27.597115 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.597088 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6"] Oct 30 18:33:27.602926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.602900 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-fde2c5/simulator-w2-fde2c5-kserve-58748bf4b6-hzcs6"] Oct 30 18:33:27.710788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:27.710762 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb"] Oct 30 18:33:27.712479 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:33:27.712453 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeecf0c23_83bb_42aa_bd59_18d9dae718e2.slice/crio-fe0e36aa3050c5ae749f156515d3a0b085b54a65f78a6fb0f52159917d9dc7b1 WatchSource:0}: Error finding container fe0e36aa3050c5ae749f156515d3a0b085b54a65f78a6fb0f52159917d9dc7b1: Status 404 returned error can't find the container with id fe0e36aa3050c5ae749f156515d3a0b085b54a65f78a6fb0f52159917d9dc7b1 Oct 30 18:33:28.521992 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:28.521959 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" event={"ID":"eecf0c23-83bb-42aa-bd59-18d9dae718e2","Type":"ContainerStarted","Data":"e5d7f1164af3e13b2ecf4a2d874fefcc30e83e28faedd7e7916450d519945fbc"} Oct 30 18:33:28.522202 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:28.522013 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" event={"ID":"eecf0c23-83bb-42aa-bd59-18d9dae718e2","Type":"ContainerStarted","Data":"fe0e36aa3050c5ae749f156515d3a0b085b54a65f78a6fb0f52159917d9dc7b1"} Oct 30 18:33:28.559657 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:28.559599 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" podStartSLOduration=1.559585079 podStartE2EDuration="1.559585079s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:33:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:33:28.559266555 +0000 UTC m=+1907.875686877" watchObservedRunningTime="2026-10-30 18:33:28.559585079 +0000 UTC m=+1907.876005400" Oct 30 18:33:28.801896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:28.801816 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73249ea3-3ad3-489d-ac99-b1145098e719" path="/var/lib/kubelet/pods/73249ea3-3ad3-489d-ac99-b1145098e719/volumes" Oct 30 18:33:28.802265 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:28.802210 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b" path="/var/lib/kubelet/pods/b6a6cc05-a0c6-4ddf-bfc0-782f1f66620b/volumes" Oct 30 18:33:29.530046 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:29.530009 5232 generic.go:356] "Generic (PLEG): container finished" podID="eecf0c23-83bb-42aa-bd59-18d9dae718e2" containerID="e5d7f1164af3e13b2ecf4a2d874fefcc30e83e28faedd7e7916450d519945fbc" exitCode=0 Oct 30 18:33:29.530227 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:29.530085 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" event={"ID":"eecf0c23-83bb-42aa-bd59-18d9dae718e2","Type":"ContainerDied","Data":"e5d7f1164af3e13b2ecf4a2d874fefcc30e83e28faedd7e7916450d519945fbc"} Oct 30 18:33:30.706092 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:30.706066 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" Oct 30 18:33:30.794331 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:30.794253 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eecf0c23-83bb-42aa-bd59-18d9dae718e2-kube-api-access-9zrds\" (UniqueName: \"kubernetes.io/projected/eecf0c23-83bb-42aa-bd59-18d9dae718e2-kube-api-access-9zrds\") pod \"eecf0c23-83bb-42aa-bd59-18d9dae718e2\" (UID: \"eecf0c23-83bb-42aa-bd59-18d9dae718e2\") " Oct 30 18:33:30.794331 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:30.794318 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eecf0c23-83bb-42aa-bd59-18d9dae718e2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecf0c23-83bb-42aa-bd59-18d9dae718e2-maas-api-openshift-ca\") pod \"eecf0c23-83bb-42aa-bd59-18d9dae718e2\" (UID: \"eecf0c23-83bb-42aa-bd59-18d9dae718e2\") " Oct 30 18:33:30.794546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:30.794359 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eecf0c23-83bb-42aa-bd59-18d9dae718e2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecf0c23-83bb-42aa-bd59-18d9dae718e2-maas-api-xks-ca\") pod \"eecf0c23-83bb-42aa-bd59-18d9dae718e2\" (UID: \"eecf0c23-83bb-42aa-bd59-18d9dae718e2\") " Oct 30 18:33:30.794697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:30.794676 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eecf0c23-83bb-42aa-bd59-18d9dae718e2-maas-api-openshift-ca" pod "eecf0c23-83bb-42aa-bd59-18d9dae718e2" (UID: "eecf0c23-83bb-42aa-bd59-18d9dae718e2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:30.796794 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:30.796770 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eecf0c23-83bb-42aa-bd59-18d9dae718e2-maas-api-xks-ca" pod "eecf0c23-83bb-42aa-bd59-18d9dae718e2" (UID: "eecf0c23-83bb-42aa-bd59-18d9dae718e2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:30.796794 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:30.796776 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eecf0c23-83bb-42aa-bd59-18d9dae718e2-kube-api-access-9zrds" pod "eecf0c23-83bb-42aa-bd59-18d9dae718e2" (UID: "eecf0c23-83bb-42aa-bd59-18d9dae718e2"). InnerVolumeSpecName "kube-api-access-9zrds". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:30.895426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:30.895390 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9zrds\" (UniqueName: \"kubernetes.io/projected/eecf0c23-83bb-42aa-bd59-18d9dae718e2-kube-api-access-9zrds\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:30.895426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:30.895422 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eecf0c23-83bb-42aa-bd59-18d9dae718e2-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:30.895426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:30.895433 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eecf0c23-83bb-42aa-bd59-18d9dae718e2-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:30.913829 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:30.913794 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" podUID="1f3f54d2-9382-4abd-81a5-4fba85c28dad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:33:31.026885 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.026848 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689"] Oct 30 18:33:31.029033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.029010 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fe1ddc9-hd689"] Oct 30 18:33:31.545140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.545114 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" Oct 30 18:33:31.545140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.545126 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" event={"ID":"eecf0c23-83bb-42aa-bd59-18d9dae718e2","Type":"ContainerDied","Data":"fe0e36aa3050c5ae749f156515d3a0b085b54a65f78a6fb0f52159917d9dc7b1"} Oct 30 18:33:31.545323 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.545189 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fe0e36aa3050c5ae749f156515d3a0b085b54a65f78a6fb0f52159917d9dc7b1" Oct 30 18:33:31.740699 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.740671 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:33:31.804693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.804608 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b16e19a-e62c-47b4-b695-ccc7164524b5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b16e19a-e62c-47b4-b695-ccc7164524b5-istiod-ca-cert\") pod \"7b16e19a-e62c-47b4-b695-ccc7164524b5\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " Oct 30 18:33:31.804693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.804676 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-workload-socket\") pod \"7b16e19a-e62c-47b4-b695-ccc7164524b5\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " Oct 30 18:33:31.804919 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.804717 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-token\" (UniqueName: \"kubernetes.io/projected/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-token\") pod \"7b16e19a-e62c-47b4-b695-ccc7164524b5\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " Oct 30 18:33:31.804919 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.804787 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-workload-certs\") pod \"7b16e19a-e62c-47b4-b695-ccc7164524b5\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " Oct 30 18:33:31.804919 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.804829 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-data\") pod \"7b16e19a-e62c-47b4-b695-ccc7164524b5\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " Oct 30 18:33:31.805082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.804927 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-envoy\") pod \"7b16e19a-e62c-47b4-b695-ccc7164524b5\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " Oct 30 18:33:31.805082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.804955 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-podinfo\") pod \"7b16e19a-e62c-47b4-b695-ccc7164524b5\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " Oct 30 18:33:31.805082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.804967 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-workload-socket" pod "7b16e19a-e62c-47b4-b695-ccc7164524b5" (UID: "7b16e19a-e62c-47b4-b695-ccc7164524b5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:31.805082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.805045 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b16e19a-e62c-47b4-b695-ccc7164524b5-kube-api-access-tc2z6\" (UniqueName: \"kubernetes.io/projected/7b16e19a-e62c-47b4-b695-ccc7164524b5-kube-api-access-tc2z6\") pod \"7b16e19a-e62c-47b4-b695-ccc7164524b5\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " Oct 30 18:33:31.805082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.805077 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-credential-socket\") pod \"7b16e19a-e62c-47b4-b695-ccc7164524b5\" (UID: \"7b16e19a-e62c-47b4-b695-ccc7164524b5\") " Oct 30 18:33:31.805460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.805140 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b16e19a-e62c-47b4-b695-ccc7164524b5-istiod-ca-cert" pod "7b16e19a-e62c-47b4-b695-ccc7164524b5" (UID: "7b16e19a-e62c-47b4-b695-ccc7164524b5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:31.805460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.805169 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-data" pod "7b16e19a-e62c-47b4-b695-ccc7164524b5" (UID: "7b16e19a-e62c-47b4-b695-ccc7164524b5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:31.805460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.805399 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-workload-certs" pod "7b16e19a-e62c-47b4-b695-ccc7164524b5" (UID: "7b16e19a-e62c-47b4-b695-ccc7164524b5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:31.805821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.805473 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/7b16e19a-e62c-47b4-b695-ccc7164524b5-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:31.805821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.805528 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:31.805821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.805543 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:31.805821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.805662 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-credential-socket" pod "7b16e19a-e62c-47b4-b695-ccc7164524b5" (UID: "7b16e19a-e62c-47b4-b695-ccc7164524b5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:31.807263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.807237 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-token" pod "7b16e19a-e62c-47b4-b695-ccc7164524b5" (UID: "7b16e19a-e62c-47b4-b695-ccc7164524b5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:31.807414 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.807390 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-envoy" pod "7b16e19a-e62c-47b4-b695-ccc7164524b5" (UID: "7b16e19a-e62c-47b4-b695-ccc7164524b5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:31.807919 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.807895 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-podinfo" pod "7b16e19a-e62c-47b4-b695-ccc7164524b5" (UID: "7b16e19a-e62c-47b4-b695-ccc7164524b5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:33:31.808034 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.808009 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b16e19a-e62c-47b4-b695-ccc7164524b5-kube-api-access-tc2z6" pod "7b16e19a-e62c-47b4-b695-ccc7164524b5" (UID: "7b16e19a-e62c-47b4-b695-ccc7164524b5"). InnerVolumeSpecName "kube-api-access-tc2z6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:31.906852 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.906810 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:31.906852 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.906844 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:31.906852 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.906853 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:31.906852 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.906861 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tc2z6\" (UniqueName: \"kubernetes.io/projected/7b16e19a-e62c-47b4-b695-ccc7164524b5-kube-api-access-tc2z6\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:31.907136 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.906871 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/7b16e19a-e62c-47b4-b695-ccc7164524b5-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:31.907136 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.906880 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/7b16e19a-e62c-47b4-b695-ccc7164524b5-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:31.921962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:31.921926 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" podUID="048571d7-450a-40b4-8c18-94ec10221b86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:33:32.554257 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:32.554226 5232 generic.go:356] "Generic (PLEG): container finished" podID="7b16e19a-e62c-47b4-b695-ccc7164524b5" containerID="887adb3ce5d62083c4399327087dabba8ad5131dfda0bd3082011f56093cc782" exitCode=0 Oct 30 18:33:32.554445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:32.554282 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" Oct 30 18:33:32.554445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:32.554289 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" event={"ID":"7b16e19a-e62c-47b4-b695-ccc7164524b5","Type":"ContainerDied","Data":"887adb3ce5d62083c4399327087dabba8ad5131dfda0bd3082011f56093cc782"} Oct 30 18:33:32.554445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:32.554342 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2" event={"ID":"7b16e19a-e62c-47b4-b695-ccc7164524b5","Type":"ContainerDied","Data":"6a824ebad85a21e9311e7f5658ac542e571ae1756d9feebd6839e7a2948c704f"} Oct 30 18:33:32.554445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:32.554372 5232 scope.go:122] "RemoveContainer" containerID="887adb3ce5d62083c4399327087dabba8ad5131dfda0bd3082011f56093cc782" Oct 30 18:33:32.565387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:32.565367 5232 scope.go:122] "RemoveContainer" containerID="dc88d7c05e156b2e31e11cdf0e8850164c98e39a3b42d279cb811095a855ec7d" Oct 30 18:33:32.574020 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:32.574003 5232 scope.go:122] "RemoveContainer" containerID="887adb3ce5d62083c4399327087dabba8ad5131dfda0bd3082011f56093cc782" Oct 30 18:33:32.574280 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:33:32.574261 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"887adb3ce5d62083c4399327087dabba8ad5131dfda0bd3082011f56093cc782\": container with ID starting with 887adb3ce5d62083c4399327087dabba8ad5131dfda0bd3082011f56093cc782 not found: ID does not exist" containerID="887adb3ce5d62083c4399327087dabba8ad5131dfda0bd3082011f56093cc782" Oct 30 18:33:32.574330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:32.574289 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"887adb3ce5d62083c4399327087dabba8ad5131dfda0bd3082011f56093cc782"} err="failed to get container status \"887adb3ce5d62083c4399327087dabba8ad5131dfda0bd3082011f56093cc782\": rpc error: code = NotFound desc = could not find container \"887adb3ce5d62083c4399327087dabba8ad5131dfda0bd3082011f56093cc782\": container with ID starting with 887adb3ce5d62083c4399327087dabba8ad5131dfda0bd3082011f56093cc782 not found: ID does not exist" Oct 30 18:33:32.574330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:32.574309 5232 scope.go:122] "RemoveContainer" containerID="dc88d7c05e156b2e31e11cdf0e8850164c98e39a3b42d279cb811095a855ec7d" Oct 30 18:33:32.574552 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:33:32.574534 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc88d7c05e156b2e31e11cdf0e8850164c98e39a3b42d279cb811095a855ec7d\": container with ID starting with dc88d7c05e156b2e31e11cdf0e8850164c98e39a3b42d279cb811095a855ec7d not found: ID does not exist" containerID="dc88d7c05e156b2e31e11cdf0e8850164c98e39a3b42d279cb811095a855ec7d" Oct 30 18:33:32.574606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:32.574557 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc88d7c05e156b2e31e11cdf0e8850164c98e39a3b42d279cb811095a855ec7d"} err="failed to get container status \"dc88d7c05e156b2e31e11cdf0e8850164c98e39a3b42d279cb811095a855ec7d\": rpc error: code = NotFound desc = could not find container \"dc88d7c05e156b2e31e11cdf0e8850164c98e39a3b42d279cb811095a855ec7d\": container with ID starting with dc88d7c05e156b2e31e11cdf0e8850164c98e39a3b42d279cb811095a855ec7d not found: ID does not exist" Oct 30 18:33:32.604764 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:32.604735 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2"] Oct 30 18:33:32.617160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:32.617130 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w4-c22995-openshift-default-769448f675-2m8r2"] Oct 30 18:33:32.801739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:32.801706 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0cdcd6ec-0417-4dd7-8ca7-032c12b859e3" path="/var/lib/kubelet/pods/0cdcd6ec-0417-4dd7-8ca7-032c12b859e3/volumes" Oct 30 18:33:32.802159 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:32.802144 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b16e19a-e62c-47b4-b695-ccc7164524b5" path="/var/lib/kubelet/pods/7b16e19a-e62c-47b4-b695-ccc7164524b5/volumes" Oct 30 18:33:37.576402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:37.576339 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb"] Oct 30 18:33:37.608600 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:37.608570 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh"] Oct 30 18:33:40.793398 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:40.793365 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb"] Oct 30 18:33:40.805163 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:40.805134 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb"] Oct 30 18:33:40.811817 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:33:40.811773 5232 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb\" is forbidden: User \"system:node:ip-10-0-141-106.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-141-106.ec2.internal' and this object" podUID="eecf0c23-83bb-42aa-bd59-18d9dae718e2" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-f0837a-smjgb" Oct 30 18:33:40.922983 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:40.922946 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" podUID="1f3f54d2-9382-4abd-81a5-4fba85c28dad" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:33:41.352373 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:41.352338 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s"] Oct 30 18:33:41.364464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:41.364437 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-fde2c5-jn57s"] Oct 30 18:33:41.935447 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:41.935397 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" podUID="048571d7-450a-40b4-8c18-94ec10221b86" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:33:42.041912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.041870 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95"] Oct 30 18:33:42.049612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.049584 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-30ae5c7c-hfh95"] Oct 30 18:33:42.307089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.307053 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" Oct 30 18:33:42.333533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.325743 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" Oct 30 18:33:42.461103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.461018 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/048571d7-450a-40b4-8c18-94ec10221b86-kube-api-access-w6dd4\" (UniqueName: \"kubernetes.io/projected/048571d7-450a-40b4-8c18-94ec10221b86-kube-api-access-w6dd4\") pod \"048571d7-450a-40b4-8c18-94ec10221b86\" (UID: \"048571d7-450a-40b4-8c18-94ec10221b86\") " Oct 30 18:33:42.461103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.461075 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1f3f54d2-9382-4abd-81a5-4fba85c28dad-kube-api-access-8p9lp\" (UniqueName: \"kubernetes.io/projected/1f3f54d2-9382-4abd-81a5-4fba85c28dad-kube-api-access-8p9lp\") pod \"1f3f54d2-9382-4abd-81a5-4fba85c28dad\" (UID: \"1f3f54d2-9382-4abd-81a5-4fba85c28dad\") " Oct 30 18:33:42.461360 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.461127 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/048571d7-450a-40b4-8c18-94ec10221b86-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048571d7-450a-40b4-8c18-94ec10221b86-plugins-config-volume\") pod \"048571d7-450a-40b4-8c18-94ec10221b86\" (UID: \"048571d7-450a-40b4-8c18-94ec10221b86\") " Oct 30 18:33:42.461360 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.461187 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1f3f54d2-9382-4abd-81a5-4fba85c28dad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3f54d2-9382-4abd-81a5-4fba85c28dad-plugins-config-volume\") pod \"1f3f54d2-9382-4abd-81a5-4fba85c28dad\" (UID: \"1f3f54d2-9382-4abd-81a5-4fba85c28dad\") " Oct 30 18:33:42.461761 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.461702 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1f3f54d2-9382-4abd-81a5-4fba85c28dad-plugins-config-volume" pod "1f3f54d2-9382-4abd-81a5-4fba85c28dad" (UID: "1f3f54d2-9382-4abd-81a5-4fba85c28dad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:42.461863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.461798 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/048571d7-450a-40b4-8c18-94ec10221b86-plugins-config-volume" pod "048571d7-450a-40b4-8c18-94ec10221b86" (UID: "048571d7-450a-40b4-8c18-94ec10221b86"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:42.463995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.463971 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/048571d7-450a-40b4-8c18-94ec10221b86-kube-api-access-w6dd4" pod "048571d7-450a-40b4-8c18-94ec10221b86" (UID: "048571d7-450a-40b4-8c18-94ec10221b86"). InnerVolumeSpecName "kube-api-access-w6dd4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:42.464105 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.464004 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1f3f54d2-9382-4abd-81a5-4fba85c28dad-kube-api-access-8p9lp" pod "1f3f54d2-9382-4abd-81a5-4fba85c28dad" (UID: "1f3f54d2-9382-4abd-81a5-4fba85c28dad"). InnerVolumeSpecName "kube-api-access-8p9lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:42.562883 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.562852 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1f3f54d2-9382-4abd-81a5-4fba85c28dad-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:42.562883 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.562880 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6dd4\" (UniqueName: \"kubernetes.io/projected/048571d7-450a-40b4-8c18-94ec10221b86-kube-api-access-w6dd4\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:42.563106 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.562891 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8p9lp\" (UniqueName: \"kubernetes.io/projected/1f3f54d2-9382-4abd-81a5-4fba85c28dad-kube-api-access-8p9lp\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:42.563106 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.562902 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/048571d7-450a-40b4-8c18-94ec10221b86-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:42.577589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.577475 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb" podUID="ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b" containerName="payload-processing" containerID="cri-o://bcf0d99a0e043328883f081b2c8e83836a1746e488e8a5549f479011168aa723" gracePeriod=25 Oct 30 18:33:42.609296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.609254 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh" podUID="2ca962a0-f12b-43a2-994e-43a21e407dd2" containerName="payload-pre-processing" containerID="cri-o://8a4220c23b4a9f9ecf4279c14f1d1c2e8a87a47d8de57ade9e384829c43ce22d" gracePeriod=25 Oct 30 18:33:42.642814 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.642772 5232 generic.go:356] "Generic (PLEG): container finished" podID="048571d7-450a-40b4-8c18-94ec10221b86" containerID="2321ac6555ed9e7b126d2a02fc840331f148df60159a950adb5261d090c50ca8" exitCode=0 Oct 30 18:33:42.642972 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.642820 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" Oct 30 18:33:42.642972 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.642843 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" event={"ID":"048571d7-450a-40b4-8c18-94ec10221b86","Type":"ContainerDied","Data":"2321ac6555ed9e7b126d2a02fc840331f148df60159a950adb5261d090c50ca8"} Oct 30 18:33:42.642972 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.642925 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7" event={"ID":"048571d7-450a-40b4-8c18-94ec10221b86","Type":"ContainerDied","Data":"cc896570eb90dfc3b2245d7b5d0b1439c4392d69dda0c85c208e749ce7eaec08"} Oct 30 18:33:42.642972 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.642950 5232 scope.go:122] "RemoveContainer" containerID="2321ac6555ed9e7b126d2a02fc840331f148df60159a950adb5261d090c50ca8" Oct 30 18:33:42.644956 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.644934 5232 generic.go:356] "Generic (PLEG): container finished" podID="1f3f54d2-9382-4abd-81a5-4fba85c28dad" containerID="b6f8d0ddbb6dad0de774503616b83ab4ec48d5c7c6d3c37dcf9a0d34f7558eea" exitCode=0 Oct 30 18:33:42.645072 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.644986 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" Oct 30 18:33:42.645334 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.644987 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" event={"ID":"1f3f54d2-9382-4abd-81a5-4fba85c28dad","Type":"ContainerDied","Data":"b6f8d0ddbb6dad0de774503616b83ab4ec48d5c7c6d3c37dcf9a0d34f7558eea"} Oct 30 18:33:42.645401 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.645359 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl" event={"ID":"1f3f54d2-9382-4abd-81a5-4fba85c28dad","Type":"ContainerDied","Data":"22e2954ac93f5cd2cc2863badb7f31cfa269163903a84e1153c595b5003aebee"} Oct 30 18:33:42.745109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.745083 5232 scope.go:122] "RemoveContainer" containerID="2321ac6555ed9e7b126d2a02fc840331f148df60159a950adb5261d090c50ca8" Oct 30 18:33:42.745595 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:33:42.745496 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2321ac6555ed9e7b126d2a02fc840331f148df60159a950adb5261d090c50ca8\": container with ID starting with 2321ac6555ed9e7b126d2a02fc840331f148df60159a950adb5261d090c50ca8 not found: ID does not exist" containerID="2321ac6555ed9e7b126d2a02fc840331f148df60159a950adb5261d090c50ca8" Oct 30 18:33:42.745716 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.745605 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2321ac6555ed9e7b126d2a02fc840331f148df60159a950adb5261d090c50ca8"} err="failed to get container status \"2321ac6555ed9e7b126d2a02fc840331f148df60159a950adb5261d090c50ca8\": rpc error: code = NotFound desc = could not find container \"2321ac6555ed9e7b126d2a02fc840331f148df60159a950adb5261d090c50ca8\": container with ID starting with 2321ac6555ed9e7b126d2a02fc840331f148df60159a950adb5261d090c50ca8 not found: ID does not exist" Oct 30 18:33:42.745716 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.745624 5232 scope.go:122] "RemoveContainer" containerID="b6f8d0ddbb6dad0de774503616b83ab4ec48d5c7c6d3c37dcf9a0d34f7558eea" Oct 30 18:33:42.747219 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.747194 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7"] Oct 30 18:33:42.757812 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.757792 5232 scope.go:122] "RemoveContainer" containerID="b6f8d0ddbb6dad0de774503616b83ab4ec48d5c7c6d3c37dcf9a0d34f7558eea" Oct 30 18:33:42.758283 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:33:42.758199 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b6f8d0ddbb6dad0de774503616b83ab4ec48d5c7c6d3c37dcf9a0d34f7558eea\": container with ID starting with b6f8d0ddbb6dad0de774503616b83ab4ec48d5c7c6d3c37dcf9a0d34f7558eea not found: ID does not exist" containerID="b6f8d0ddbb6dad0de774503616b83ab4ec48d5c7c6d3c37dcf9a0d34f7558eea" Oct 30 18:33:42.758283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.758239 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b6f8d0ddbb6dad0de774503616b83ab4ec48d5c7c6d3c37dcf9a0d34f7558eea"} err="failed to get container status \"b6f8d0ddbb6dad0de774503616b83ab4ec48d5c7c6d3c37dcf9a0d34f7558eea\": rpc error: code = NotFound desc = could not find container \"b6f8d0ddbb6dad0de774503616b83ab4ec48d5c7c6d3c37dcf9a0d34f7558eea\": container with ID starting with b6f8d0ddbb6dad0de774503616b83ab4ec48d5c7c6d3c37dcf9a0d34f7558eea not found: ID does not exist" Oct 30 18:33:42.759334 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.759302 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-fde2c5-76f6995c88-99qv7"] Oct 30 18:33:42.783002 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.782968 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl"] Oct 30 18:33:42.786130 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.786106 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-fde2c5-7546567987-rsgkl"] Oct 30 18:33:42.804804 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.804534 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="048571d7-450a-40b4-8c18-94ec10221b86" path="/var/lib/kubelet/pods/048571d7-450a-40b4-8c18-94ec10221b86/volumes" Oct 30 18:33:42.805070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.805039 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1f3f54d2-9382-4abd-81a5-4fba85c28dad" path="/var/lib/kubelet/pods/1f3f54d2-9382-4abd-81a5-4fba85c28dad/volumes" Oct 30 18:33:42.805489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.805463 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2696076d-3cc9-4859-bf02-25ab6c93ce79" path="/var/lib/kubelet/pods/2696076d-3cc9-4859-bf02-25ab6c93ce79/volumes" Oct 30 18:33:42.806737 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.806713 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf57d029-4ffe-44e4-b025-f28c296405b2" path="/var/lib/kubelet/pods/cf57d029-4ffe-44e4-b025-f28c296405b2/volumes" Oct 30 18:33:42.807126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.807111 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eecf0c23-83bb-42aa-bd59-18d9dae718e2" path="/var/lib/kubelet/pods/eecf0c23-83bb-42aa-bd59-18d9dae718e2/volumes" Oct 30 18:33:42.895340 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.895308 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb" Oct 30 18:33:42.978200 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:42.978169 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh" Oct 30 18:33:43.068076 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.067993 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b-kube-api-access-txwhl\" (UniqueName: \"kubernetes.io/projected/ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b-kube-api-access-txwhl\") pod \"ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b\" (UID: \"ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b\") " Oct 30 18:33:43.068364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.068340 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ca962a0-f12b-43a2-994e-43a21e407dd2-kube-api-access-cnzkh\" (UniqueName: \"kubernetes.io/projected/2ca962a0-f12b-43a2-994e-43a21e407dd2-kube-api-access-cnzkh\") pod \"2ca962a0-f12b-43a2-994e-43a21e407dd2\" (UID: \"2ca962a0-f12b-43a2-994e-43a21e407dd2\") " Oct 30 18:33:43.068596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.068574 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b-plugins-config-volume\") pod \"ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b\" (UID: \"ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b\") " Oct 30 18:33:43.068864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.068844 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ca962a0-f12b-43a2-994e-43a21e407dd2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca962a0-f12b-43a2-994e-43a21e407dd2-plugins-config-volume\") pod \"2ca962a0-f12b-43a2-994e-43a21e407dd2\" (UID: \"2ca962a0-f12b-43a2-994e-43a21e407dd2\") " Oct 30 18:33:43.069301 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.068980 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b-plugins-config-volume" pod "ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b" (UID: "ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:43.069437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.069416 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ca962a0-f12b-43a2-994e-43a21e407dd2-plugins-config-volume" pod "2ca962a0-f12b-43a2-994e-43a21e407dd2" (UID: "2ca962a0-f12b-43a2-994e-43a21e407dd2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:43.069794 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.069776 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2ca962a0-f12b-43a2-994e-43a21e407dd2-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:43.069971 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.069916 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:43.071978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.071946 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b-kube-api-access-txwhl" pod "ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b" (UID: "ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b"). InnerVolumeSpecName "kube-api-access-txwhl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:43.072864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.072837 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ca962a0-f12b-43a2-994e-43a21e407dd2-kube-api-access-cnzkh" pod "2ca962a0-f12b-43a2-994e-43a21e407dd2" (UID: "2ca962a0-f12b-43a2-994e-43a21e407dd2"). InnerVolumeSpecName "kube-api-access-cnzkh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:43.171152 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.171115 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-txwhl\" (UniqueName: \"kubernetes.io/projected/ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b-kube-api-access-txwhl\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:43.171152 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.171146 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cnzkh\" (UniqueName: \"kubernetes.io/projected/2ca962a0-f12b-43a2-994e-43a21e407dd2-kube-api-access-cnzkh\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:43.652380 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.652350 5232 generic.go:356] "Generic (PLEG): container finished" podID="ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b" containerID="bcf0d99a0e043328883f081b2c8e83836a1746e488e8a5549f479011168aa723" exitCode=0 Oct 30 18:33:43.652699 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.652436 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb" event={"ID":"ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b","Type":"ContainerDied","Data":"bcf0d99a0e043328883f081b2c8e83836a1746e488e8a5549f479011168aa723"} Oct 30 18:33:43.652807 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.652731 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb" event={"ID":"ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b","Type":"ContainerDied","Data":"9f47c464d5b026d969405117ba6bedec2c19a70a16f6fade4cd9bb688e8a8a19"} Oct 30 18:33:43.652807 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.652761 5232 scope.go:122] "RemoveContainer" containerID="bcf0d99a0e043328883f081b2c8e83836a1746e488e8a5549f479011168aa723" Oct 30 18:33:43.652807 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.652447 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb" Oct 30 18:33:43.661969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.661944 5232 generic.go:356] "Generic (PLEG): container finished" podID="2ca962a0-f12b-43a2-994e-43a21e407dd2" containerID="8a4220c23b4a9f9ecf4279c14f1d1c2e8a87a47d8de57ade9e384829c43ce22d" exitCode=0 Oct 30 18:33:43.662082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.661989 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh" event={"ID":"2ca962a0-f12b-43a2-994e-43a21e407dd2","Type":"ContainerDied","Data":"8a4220c23b4a9f9ecf4279c14f1d1c2e8a87a47d8de57ade9e384829c43ce22d"} Oct 30 18:33:43.662082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.662030 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh" event={"ID":"2ca962a0-f12b-43a2-994e-43a21e407dd2","Type":"ContainerDied","Data":"2200810e1447b954bfbfd4fafe4ed345c57efdc6a24cf818c3dd16226ed2967c"} Oct 30 18:33:43.662304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.662113 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh" Oct 30 18:33:43.671594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.671570 5232 scope.go:122] "RemoveContainer" containerID="bcf0d99a0e043328883f081b2c8e83836a1746e488e8a5549f479011168aa723" Oct 30 18:33:43.671953 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:33:43.671928 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bcf0d99a0e043328883f081b2c8e83836a1746e488e8a5549f479011168aa723\": container with ID starting with bcf0d99a0e043328883f081b2c8e83836a1746e488e8a5549f479011168aa723 not found: ID does not exist" containerID="bcf0d99a0e043328883f081b2c8e83836a1746e488e8a5549f479011168aa723" Oct 30 18:33:43.672059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.671961 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bcf0d99a0e043328883f081b2c8e83836a1746e488e8a5549f479011168aa723"} err="failed to get container status \"bcf0d99a0e043328883f081b2c8e83836a1746e488e8a5549f479011168aa723\": rpc error: code = NotFound desc = could not find container \"bcf0d99a0e043328883f081b2c8e83836a1746e488e8a5549f479011168aa723\": container with ID starting with bcf0d99a0e043328883f081b2c8e83836a1746e488e8a5549f479011168aa723 not found: ID does not exist" Oct 30 18:33:43.672059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.671977 5232 scope.go:122] "RemoveContainer" containerID="8a4220c23b4a9f9ecf4279c14f1d1c2e8a87a47d8de57ade9e384829c43ce22d" Oct 30 18:33:43.681297 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.681273 5232 scope.go:122] "RemoveContainer" containerID="8a4220c23b4a9f9ecf4279c14f1d1c2e8a87a47d8de57ade9e384829c43ce22d" Oct 30 18:33:43.681616 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:33:43.681592 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a4220c23b4a9f9ecf4279c14f1d1c2e8a87a47d8de57ade9e384829c43ce22d\": container with ID starting with 8a4220c23b4a9f9ecf4279c14f1d1c2e8a87a47d8de57ade9e384829c43ce22d not found: ID does not exist" containerID="8a4220c23b4a9f9ecf4279c14f1d1c2e8a87a47d8de57ade9e384829c43ce22d" Oct 30 18:33:43.681678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.681630 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a4220c23b4a9f9ecf4279c14f1d1c2e8a87a47d8de57ade9e384829c43ce22d"} err="failed to get container status \"8a4220c23b4a9f9ecf4279c14f1d1c2e8a87a47d8de57ade9e384829c43ce22d\": rpc error: code = NotFound desc = could not find container \"8a4220c23b4a9f9ecf4279c14f1d1c2e8a87a47d8de57ade9e384829c43ce22d\": container with ID starting with 8a4220c23b4a9f9ecf4279c14f1d1c2e8a87a47d8de57ade9e384829c43ce22d not found: ID does not exist" Oct 30 18:33:43.707761 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.707725 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb"] Oct 30 18:33:43.724540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.723251 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-f0837a-594bb5fb-ldrdb"] Oct 30 18:33:43.761780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.761729 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh"] Oct 30 18:33:43.765695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:43.765660 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w4-f0837a-69bbbd4b8c-7pcnh"] Oct 30 18:33:44.801663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:44.801627 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ca962a0-f12b-43a2-994e-43a21e407dd2" path="/var/lib/kubelet/pods/2ca962a0-f12b-43a2-994e-43a21e407dd2/volumes" Oct 30 18:33:44.802215 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:44.802162 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b" path="/var/lib/kubelet/pods/ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b/volumes" Oct 30 18:33:47.698137 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:47.698045 5232 generic.go:356] "Generic (PLEG): container finished" podID="c5711415-04ea-4954-a672-2af1986aec4c" containerID="e274ceddbbde51399c65f2b46e78a8132cec3ee8400152bd462e8728ea7cc3fe" exitCode=0 Oct 30 18:33:47.698629 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:47.698244 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" event={"ID":"c5711415-04ea-4954-a672-2af1986aec4c","Type":"ContainerDied","Data":"e274ceddbbde51399c65f2b46e78a8132cec3ee8400152bd462e8728ea7cc3fe"} Oct 30 18:33:47.698764 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:47.698742 5232 scope.go:122] "RemoveContainer" containerID="e274ceddbbde51399c65f2b46e78a8132cec3ee8400152bd462e8728ea7cc3fe" Oct 30 18:33:47.700073 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:47.700041 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:33:47.700156 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:47.700113 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:33:47.700156 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:47.700145 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:33:48.009613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.009580 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m"] Oct 30 18:33:48.706391 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.706348 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" event={"ID":"c5711415-04ea-4954-a672-2af1986aec4c","Type":"ContainerStarted","Data":"186aa2b278648537cd6e2a6dac2b132f3d30d7f608cc4864f7b9fc87eebd9149"} Oct 30 18:33:48.706876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.706440 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" podUID="c5711415-04ea-4954-a672-2af1986aec4c" containerName="istio-proxy" containerID="cri-o://186aa2b278648537cd6e2a6dac2b132f3d30d7f608cc4864f7b9fc87eebd9149" gracePeriod=30 Oct 30 18:33:48.761088 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.761052 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c"] Oct 30 18:33:48.761914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.761675 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7b16e19a-e62c-47b4-b695-ccc7164524b5" containerName="istio-proxy" Oct 30 18:33:48.761914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.761706 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b16e19a-e62c-47b4-b695-ccc7164524b5" containerName="istio-proxy" Oct 30 18:33:48.761914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.761740 5232 cpu_manager.go:414] "containerMap: removing container" podUID="eecf0c23-83bb-42aa-bd59-18d9dae718e2" containerName="revoke-keys" Oct 30 18:33:48.761914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.761748 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eecf0c23-83bb-42aa-bd59-18d9dae718e2" containerName="revoke-keys" Oct 30 18:33:48.761914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.761762 5232 cpu_manager.go:414] "containerMap: removing container" podUID="ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b" containerName="payload-processing" Oct 30 18:33:48.761914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.761769 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b" containerName="payload-processing" Oct 30 18:33:48.762252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.762163 5232 cpu_manager.go:414] "containerMap: removing container" podUID="1f3f54d2-9382-4abd-81a5-4fba85c28dad" containerName="payload-pre-processing" Oct 30 18:33:48.762252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.762181 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1f3f54d2-9382-4abd-81a5-4fba85c28dad" containerName="payload-pre-processing" Oct 30 18:33:48.762252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.762196 5232 cpu_manager.go:414] "containerMap: removing container" podUID="048571d7-450a-40b4-8c18-94ec10221b86" containerName="payload-processing" Oct 30 18:33:48.762252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.762205 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="048571d7-450a-40b4-8c18-94ec10221b86" containerName="payload-processing" Oct 30 18:33:48.762252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.762216 5232 cpu_manager.go:414] "containerMap: removing container" podUID="2ca962a0-f12b-43a2-994e-43a21e407dd2" containerName="payload-pre-processing" Oct 30 18:33:48.762252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.762221 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ca962a0-f12b-43a2-994e-43a21e407dd2" containerName="payload-pre-processing" Oct 30 18:33:48.762442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.762296 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="1f3f54d2-9382-4abd-81a5-4fba85c28dad" containerName="payload-pre-processing" Oct 30 18:33:48.762442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.762306 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="048571d7-450a-40b4-8c18-94ec10221b86" containerName="payload-processing" Oct 30 18:33:48.762442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.762311 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ca962a0-f12b-43a2-994e-43a21e407dd2" containerName="payload-pre-processing" Oct 30 18:33:48.762442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.762316 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b16e19a-e62c-47b4-b695-ccc7164524b5" containerName="istio-proxy" Oct 30 18:33:48.762442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.762322 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b16e19a-e62c-47b4-b695-ccc7164524b5" containerName="istio-proxy" Oct 30 18:33:48.762442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.762329 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="eecf0c23-83bb-42aa-bd59-18d9dae718e2" containerName="revoke-keys" Oct 30 18:33:48.762442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.762335 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec1e9fc9-981d-48fc-8f19-6e8dcdc06d3b" containerName="payload-processing" Oct 30 18:33:48.764386 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.764370 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c" Oct 30 18:33:48.779720 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.779689 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:33:48.779870 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.779742 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:33:48.782351 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.782325 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c"] Oct 30 18:33:48.841204 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.841160 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c\" (UID: \"7b1d7063-78cf-4a19-b2d0-e1f42d98dc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c" Oct 30 18:33:48.841407 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.841224 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wmvvj\" (UniqueName: \"kubernetes.io/projected/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-kube-api-access-wmvvj\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c\" (UID: \"7b1d7063-78cf-4a19-b2d0-e1f42d98dc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c" Oct 30 18:33:48.841407 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.841381 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c\" (UID: \"7b1d7063-78cf-4a19-b2d0-e1f42d98dc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c" Oct 30 18:33:48.942479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.942442 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c\" (UID: \"7b1d7063-78cf-4a19-b2d0-e1f42d98dc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c" Oct 30 18:33:48.942688 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.942561 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c\" (UID: \"7b1d7063-78cf-4a19-b2d0-e1f42d98dc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c" Oct 30 18:33:48.942688 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.942586 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wmvvj\" (UniqueName: \"kubernetes.io/projected/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-kube-api-access-wmvvj\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c\" (UID: \"7b1d7063-78cf-4a19-b2d0-e1f42d98dc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c" Oct 30 18:33:48.943245 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.943223 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c\" (UID: \"7b1d7063-78cf-4a19-b2d0-e1f42d98dc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c" Oct 30 18:33:48.945093 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.945072 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c\" (UID: \"7b1d7063-78cf-4a19-b2d0-e1f42d98dc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c" Oct 30 18:33:48.961647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:48.961580 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wmvvj\" (UniqueName: \"kubernetes.io/projected/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-kube-api-access-wmvvj\") pod \"maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c\" (UID: \"7b1d7063-78cf-4a19-b2d0-e1f42d98dc73\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c" Oct 30 18:33:49.076916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:49.076883 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c" Oct 30 18:33:49.389197 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:49.388950 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c"] Oct 30 18:33:49.391884 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:33:49.391852 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b1d7063_78cf_4a19_b2d0_e1f42d98dc73.slice/crio-4f67502c7f019b67ab55e8cf72a756f263bbf77bc308beb21a4fc045e6acf818 WatchSource:0}: Error finding container 4f67502c7f019b67ab55e8cf72a756f263bbf77bc308beb21a4fc045e6acf818: Status 404 returned error can't find the container with id 4f67502c7f019b67ab55e8cf72a756f263bbf77bc308beb21a4fc045e6acf818 Oct 30 18:33:49.639711 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:49.639634 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992"] Oct 30 18:33:49.639863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:49.639844 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" podUID="3480923d-8625-430e-819c-628e203c4946" containerName="istio-proxy" containerID="cri-o://bb8807f96e9f65c7314888bfdbd7e39e1caa4e17e44c27525b693df398c6d627" gracePeriod=30 Oct 30 18:33:49.716210 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:49.716138 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c" event={"ID":"7b1d7063-78cf-4a19-b2d0-e1f42d98dc73","Type":"ContainerStarted","Data":"4f67502c7f019b67ab55e8cf72a756f263bbf77bc308beb21a4fc045e6acf818"} Oct 30 18:33:50.724698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:50.724606 5232 generic.go:356] "Generic (PLEG): container finished" podID="7b1d7063-78cf-4a19-b2d0-e1f42d98dc73" containerID="df666ac84f97e95aa8ecd863c473cc46d29bf1e8c0dc342467c24ccb8fa1e151" exitCode=0 Oct 30 18:33:50.725075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:50.724694 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c" event={"ID":"7b1d7063-78cf-4a19-b2d0-e1f42d98dc73","Type":"ContainerDied","Data":"df666ac84f97e95aa8ecd863c473cc46d29bf1e8c0dc342467c24ccb8fa1e151"} Oct 30 18:33:51.871531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:51.871495 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c" Oct 30 18:33:51.883596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:51.883568 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-maas-api-openshift-ca\") pod \"7b1d7063-78cf-4a19-b2d0-e1f42d98dc73\" (UID: \"7b1d7063-78cf-4a19-b2d0-e1f42d98dc73\") " Oct 30 18:33:51.883739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:51.883676 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-maas-api-xks-ca\") pod \"7b1d7063-78cf-4a19-b2d0-e1f42d98dc73\" (UID: \"7b1d7063-78cf-4a19-b2d0-e1f42d98dc73\") " Oct 30 18:33:51.883810 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:51.883790 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-kube-api-access-wmvvj\" (UniqueName: \"kubernetes.io/projected/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-kube-api-access-wmvvj\") pod \"7b1d7063-78cf-4a19-b2d0-e1f42d98dc73\" (UID: \"7b1d7063-78cf-4a19-b2d0-e1f42d98dc73\") " Oct 30 18:33:51.883962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:51.883935 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-maas-api-openshift-ca" pod "7b1d7063-78cf-4a19-b2d0-e1f42d98dc73" (UID: "7b1d7063-78cf-4a19-b2d0-e1f42d98dc73"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:51.884236 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:51.884209 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:51.885969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:51.885928 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-maas-api-xks-ca" pod "7b1d7063-78cf-4a19-b2d0-e1f42d98dc73" (UID: "7b1d7063-78cf-4a19-b2d0-e1f42d98dc73"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:33:51.886108 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:51.886066 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-kube-api-access-wmvvj" pod "7b1d7063-78cf-4a19-b2d0-e1f42d98dc73" (UID: "7b1d7063-78cf-4a19-b2d0-e1f42d98dc73"). InnerVolumeSpecName "kube-api-access-wmvvj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:51.985113 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:51.985031 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:51.985113 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:51.985061 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wmvvj\" (UniqueName: \"kubernetes.io/projected/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73-kube-api-access-wmvvj\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:52.737956 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:52.737922 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c" Oct 30 18:33:52.738135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:52.737963 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c" event={"ID":"7b1d7063-78cf-4a19-b2d0-e1f42d98dc73","Type":"ContainerDied","Data":"4f67502c7f019b67ab55e8cf72a756f263bbf77bc308beb21a4fc045e6acf818"} Oct 30 18:33:52.738135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:52.738012 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4f67502c7f019b67ab55e8cf72a756f263bbf77bc308beb21a4fc045e6acf818" Oct 30 18:33:53.466269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:53.466237 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm"] Oct 30 18:33:53.494672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:53.494638 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2"] Oct 30 18:33:53.499774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:53.499734 5232 patch_prober.go:29] interesting pod/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:33:53.499939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:53.499797 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" podUID="3480923d-8625-430e-819c-628e203c4946" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:33:53.958630 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:53.958602 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:33:54.004773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.004742 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-istio-data\") pod \"c5711415-04ea-4954-a672-2af1986aec4c\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " Oct 30 18:33:54.004955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.004808 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-workload-socket\") pod \"c5711415-04ea-4954-a672-2af1986aec4c\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " Oct 30 18:33:54.004955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.004849 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-workload-certs\") pod \"c5711415-04ea-4954-a672-2af1986aec4c\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " Oct 30 18:33:54.004955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.004940 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-credential-socket\") pod \"c5711415-04ea-4954-a672-2af1986aec4c\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " Oct 30 18:33:54.005154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.004983 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5711415-04ea-4954-a672-2af1986aec4c-kube-api-access-zfpv8\" (UniqueName: \"kubernetes.io/projected/c5711415-04ea-4954-a672-2af1986aec4c-kube-api-access-zfpv8\") pod \"c5711415-04ea-4954-a672-2af1986aec4c\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " Oct 30 18:33:54.005154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.005028 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-istio-data" pod "c5711415-04ea-4954-a672-2af1986aec4c" (UID: "c5711415-04ea-4954-a672-2af1986aec4c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:54.005154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.005053 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5711415-04ea-4954-a672-2af1986aec4c-istio-token\" (UniqueName: \"kubernetes.io/projected/c5711415-04ea-4954-a672-2af1986aec4c-istio-token\") pod \"c5711415-04ea-4954-a672-2af1986aec4c\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " Oct 30 18:33:54.005154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.005085 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5711415-04ea-4954-a672-2af1986aec4c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5711415-04ea-4954-a672-2af1986aec4c-istio-podinfo\") pod \"c5711415-04ea-4954-a672-2af1986aec4c\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " Oct 30 18:33:54.005154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.005122 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-istio-envoy\") pod \"c5711415-04ea-4954-a672-2af1986aec4c\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " Oct 30 18:33:54.005492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.005156 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5711415-04ea-4954-a672-2af1986aec4c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5711415-04ea-4954-a672-2af1986aec4c-istiod-ca-cert\") pod \"c5711415-04ea-4954-a672-2af1986aec4c\" (UID: \"c5711415-04ea-4954-a672-2af1986aec4c\") " Oct 30 18:33:54.005492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.005225 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-workload-certs" pod "c5711415-04ea-4954-a672-2af1986aec4c" (UID: "c5711415-04ea-4954-a672-2af1986aec4c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:54.005492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.005443 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-workload-socket" pod "c5711415-04ea-4954-a672-2af1986aec4c" (UID: "c5711415-04ea-4954-a672-2af1986aec4c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:54.005771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.005648 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-credential-socket" pod "c5711415-04ea-4954-a672-2af1986aec4c" (UID: "c5711415-04ea-4954-a672-2af1986aec4c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:54.005839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.005822 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:54.006046 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.005844 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:54.006046 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.005859 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:54.006046 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.005873 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:54.006046 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.005911 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5711415-04ea-4954-a672-2af1986aec4c-istiod-ca-cert" pod "c5711415-04ea-4954-a672-2af1986aec4c" (UID: "c5711415-04ea-4954-a672-2af1986aec4c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:54.008686 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.008567 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5711415-04ea-4954-a672-2af1986aec4c-istio-token" pod "c5711415-04ea-4954-a672-2af1986aec4c" (UID: "c5711415-04ea-4954-a672-2af1986aec4c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:54.008686 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.008655 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-istio-envoy" pod "c5711415-04ea-4954-a672-2af1986aec4c" (UID: "c5711415-04ea-4954-a672-2af1986aec4c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:54.008686 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.008677 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5711415-04ea-4954-a672-2af1986aec4c-istio-podinfo" pod "c5711415-04ea-4954-a672-2af1986aec4c" (UID: "c5711415-04ea-4954-a672-2af1986aec4c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:33:54.008928 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.008847 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5711415-04ea-4954-a672-2af1986aec4c-kube-api-access-zfpv8" pod "c5711415-04ea-4954-a672-2af1986aec4c" (UID: "c5711415-04ea-4954-a672-2af1986aec4c"). InnerVolumeSpecName "kube-api-access-zfpv8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:54.107139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.107105 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zfpv8\" (UniqueName: \"kubernetes.io/projected/c5711415-04ea-4954-a672-2af1986aec4c-kube-api-access-zfpv8\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:54.107139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.107134 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5711415-04ea-4954-a672-2af1986aec4c-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:54.107139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.107143 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5711415-04ea-4954-a672-2af1986aec4c-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:54.107375 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.107151 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5711415-04ea-4954-a672-2af1986aec4c-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:54.107375 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.107163 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5711415-04ea-4954-a672-2af1986aec4c-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:54.751528 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.751479 5232 generic.go:356] "Generic (PLEG): container finished" podID="c5711415-04ea-4954-a672-2af1986aec4c" containerID="186aa2b278648537cd6e2a6dac2b132f3d30d7f608cc4864f7b9fc87eebd9149" exitCode=0 Oct 30 18:33:54.751999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.751559 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" Oct 30 18:33:54.751999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.751555 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" event={"ID":"c5711415-04ea-4954-a672-2af1986aec4c","Type":"ContainerDied","Data":"186aa2b278648537cd6e2a6dac2b132f3d30d7f608cc4864f7b9fc87eebd9149"} Oct 30 18:33:54.751999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.751635 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m" event={"ID":"c5711415-04ea-4954-a672-2af1986aec4c","Type":"ContainerDied","Data":"b7405fc941451fb3dedb595667c8a5935ddbedc6ea1be3858f9ac30a60ed21ac"} Oct 30 18:33:54.751999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.751673 5232 scope.go:122] "RemoveContainer" containerID="186aa2b278648537cd6e2a6dac2b132f3d30d7f608cc4864f7b9fc87eebd9149" Oct 30 18:33:54.761873 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.761851 5232 scope.go:122] "RemoveContainer" containerID="e274ceddbbde51399c65f2b46e78a8132cec3ee8400152bd462e8728ea7cc3fe" Oct 30 18:33:54.771429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.771405 5232 scope.go:122] "RemoveContainer" containerID="186aa2b278648537cd6e2a6dac2b132f3d30d7f608cc4864f7b9fc87eebd9149" Oct 30 18:33:54.771808 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:33:54.771781 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"186aa2b278648537cd6e2a6dac2b132f3d30d7f608cc4864f7b9fc87eebd9149\": container with ID starting with 186aa2b278648537cd6e2a6dac2b132f3d30d7f608cc4864f7b9fc87eebd9149 not found: ID does not exist" containerID="186aa2b278648537cd6e2a6dac2b132f3d30d7f608cc4864f7b9fc87eebd9149" Oct 30 18:33:54.771891 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.771817 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"186aa2b278648537cd6e2a6dac2b132f3d30d7f608cc4864f7b9fc87eebd9149"} err="failed to get container status \"186aa2b278648537cd6e2a6dac2b132f3d30d7f608cc4864f7b9fc87eebd9149\": rpc error: code = NotFound desc = could not find container \"186aa2b278648537cd6e2a6dac2b132f3d30d7f608cc4864f7b9fc87eebd9149\": container with ID starting with 186aa2b278648537cd6e2a6dac2b132f3d30d7f608cc4864f7b9fc87eebd9149 not found: ID does not exist" Oct 30 18:33:54.771891 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.771836 5232 scope.go:122] "RemoveContainer" containerID="e274ceddbbde51399c65f2b46e78a8132cec3ee8400152bd462e8728ea7cc3fe" Oct 30 18:33:54.772124 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:33:54.772102 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e274ceddbbde51399c65f2b46e78a8132cec3ee8400152bd462e8728ea7cc3fe\": container with ID starting with e274ceddbbde51399c65f2b46e78a8132cec3ee8400152bd462e8728ea7cc3fe not found: ID does not exist" containerID="e274ceddbbde51399c65f2b46e78a8132cec3ee8400152bd462e8728ea7cc3fe" Oct 30 18:33:54.772167 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.772137 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e274ceddbbde51399c65f2b46e78a8132cec3ee8400152bd462e8728ea7cc3fe"} err="failed to get container status \"e274ceddbbde51399c65f2b46e78a8132cec3ee8400152bd462e8728ea7cc3fe\": rpc error: code = NotFound desc = could not find container \"e274ceddbbde51399c65f2b46e78a8132cec3ee8400152bd462e8728ea7cc3fe\": container with ID starting with e274ceddbbde51399c65f2b46e78a8132cec3ee8400152bd462e8728ea7cc3fe not found: ID does not exist" Oct 30 18:33:54.816382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.816353 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m"] Oct 30 18:33:54.827966 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.827937 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-f0837a-openshift-default-774944cc99-tf82m"] Oct 30 18:33:54.901032 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:54.900989 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:33:55.000589 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.000556 5232 scope.go:122] "RemoveContainer" containerID="5959ab3b577a1234a4b72d519e0309c13ba3e940ec36ec714e0eb7f73211ccd8" Oct 30 18:33:55.010122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.010095 5232 scope.go:122] "RemoveContainer" containerID="2b789eacda30691b99761c368abbc2e45d23be596696dfda30a2a6e4ce8b8202" Oct 30 18:33:55.014452 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.014433 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-credential-socket\") pod \"3480923d-8625-430e-819c-628e203c4946\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " Oct 30 18:33:55.014563 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.014482 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-istio-envoy\") pod \"3480923d-8625-430e-819c-628e203c4946\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " Oct 30 18:33:55.014637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.014565 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-workload-socket\") pod \"3480923d-8625-430e-819c-628e203c4946\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " Oct 30 18:33:55.014637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.014596 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3480923d-8625-430e-819c-628e203c4946-istio-token\" (UniqueName: \"kubernetes.io/projected/3480923d-8625-430e-819c-628e203c4946-istio-token\") pod \"3480923d-8625-430e-819c-628e203c4946\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " Oct 30 18:33:55.014637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.014622 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3480923d-8625-430e-819c-628e203c4946-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3480923d-8625-430e-819c-628e203c4946-istiod-ca-cert\") pod \"3480923d-8625-430e-819c-628e203c4946\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " Oct 30 18:33:55.014787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.014651 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3480923d-8625-430e-819c-628e203c4946-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3480923d-8625-430e-819c-628e203c4946-istio-podinfo\") pod \"3480923d-8625-430e-819c-628e203c4946\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " Oct 30 18:33:55.014787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.014724 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-istio-data\") pod \"3480923d-8625-430e-819c-628e203c4946\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " Oct 30 18:33:55.014787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.014766 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-workload-certs\") pod \"3480923d-8625-430e-819c-628e203c4946\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " Oct 30 18:33:55.014954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.014807 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-credential-socket" pod "3480923d-8625-430e-819c-628e203c4946" (UID: "3480923d-8625-430e-819c-628e203c4946"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:55.014954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.014839 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3480923d-8625-430e-819c-628e203c4946-kube-api-access-qgbjb\" (UniqueName: \"kubernetes.io/projected/3480923d-8625-430e-819c-628e203c4946-kube-api-access-qgbjb\") pod \"3480923d-8625-430e-819c-628e203c4946\" (UID: \"3480923d-8625-430e-819c-628e203c4946\") " Oct 30 18:33:55.014954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.014849 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-workload-socket" pod "3480923d-8625-430e-819c-628e203c4946" (UID: "3480923d-8625-430e-819c-628e203c4946"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:55.015124 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.014999 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-istio-data" pod "3480923d-8625-430e-819c-628e203c4946" (UID: "3480923d-8625-430e-819c-628e203c4946"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:55.015234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.015140 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3480923d-8625-430e-819c-628e203c4946-istiod-ca-cert" pod "3480923d-8625-430e-819c-628e203c4946" (UID: "3480923d-8625-430e-819c-628e203c4946"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:33:55.015309 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.015261 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.015309 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.015282 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.015309 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.015297 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.015444 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.015310 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3480923d-8625-430e-819c-628e203c4946-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.015590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.015566 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-workload-certs" pod "3480923d-8625-430e-819c-628e203c4946" (UID: "3480923d-8625-430e-819c-628e203c4946"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:55.017122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.017088 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-istio-envoy" pod "3480923d-8625-430e-819c-628e203c4946" (UID: "3480923d-8625-430e-819c-628e203c4946"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:33:55.017332 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.017303 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3480923d-8625-430e-819c-628e203c4946-istio-token" pod "3480923d-8625-430e-819c-628e203c4946" (UID: "3480923d-8625-430e-819c-628e203c4946"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:55.017597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.017576 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3480923d-8625-430e-819c-628e203c4946-istio-podinfo" pod "3480923d-8625-430e-819c-628e203c4946" (UID: "3480923d-8625-430e-819c-628e203c4946"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:33:55.018388 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.018367 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3480923d-8625-430e-819c-628e203c4946-kube-api-access-qgbjb" pod "3480923d-8625-430e-819c-628e203c4946" (UID: "3480923d-8625-430e-819c-628e203c4946"). InnerVolumeSpecName "kube-api-access-qgbjb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:33:55.021196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.021176 5232 scope.go:122] "RemoveContainer" containerID="bb8807f96e9f65c7314888bfdbd7e39e1caa4e17e44c27525b693df398c6d627" Oct 30 18:33:55.035335 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.035316 5232 scope.go:122] "RemoveContainer" containerID="ef21f62165cf1bd5133d84a4fa604bb51cb7c64e9b7cb45ae376d649dd3e402d" Oct 30 18:33:55.043633 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.043607 5232 scope.go:122] "RemoveContainer" containerID="3901ec1a1662a958218176f85de80b23a41c000a79110836fb227138a5e791d3" Oct 30 18:33:55.051818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.051797 5232 scope.go:122] "RemoveContainer" containerID="6f2ea9369e7c1c2866201e31b1bdaf103d68014babd6f8446bc1f9eb85df91dc" Oct 30 18:33:55.115997 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.115973 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3480923d-8625-430e-819c-628e203c4946-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.116057 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.116006 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3480923d-8625-430e-819c-628e203c4946-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.116057 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.116020 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.116137 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.116065 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qgbjb\" (UniqueName: \"kubernetes.io/projected/3480923d-8625-430e-819c-628e203c4946-kube-api-access-qgbjb\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.116137 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.116081 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3480923d-8625-430e-819c-628e203c4946-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:33:55.756953 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.756916 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" event={"ID":"3480923d-8625-430e-819c-628e203c4946","Type":"ContainerDied","Data":"bb8807f96e9f65c7314888bfdbd7e39e1caa4e17e44c27525b693df398c6d627"} Oct 30 18:33:55.757444 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.757001 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" event={"ID":"3480923d-8625-430e-819c-628e203c4946","Type":"ContainerDied","Data":"c355715b5eb8d4b875320be8ab53a1c8d014e055b22210a66c0feb7c95da26ff"} Oct 30 18:33:55.757444 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.757020 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992" Oct 30 18:33:55.848834 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.848795 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992"] Oct 30 18:33:55.850905 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:55.850883 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-fde2c5-openshift-default-5cff489bb7-s6992"] Oct 30 18:33:56.801285 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:56.801230 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3480923d-8625-430e-819c-628e203c4946" path="/var/lib/kubelet/pods/3480923d-8625-430e-819c-628e203c4946/volumes" Oct 30 18:33:56.801752 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:56.801736 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5711415-04ea-4954-a672-2af1986aec4c" path="/var/lib/kubelet/pods/c5711415-04ea-4954-a672-2af1986aec4c/volumes" Oct 30 18:33:57.033254 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:33:57.033208 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:33:59.238218 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.238182 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw"] Oct 30 18:33:59.316280 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.316233 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt"] Oct 30 18:33:59.316774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.316758 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7b16e19a-e62c-47b4-b695-ccc7164524b5" containerName="istio-proxy" Oct 30 18:33:59.316854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.316776 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b16e19a-e62c-47b4-b695-ccc7164524b5" containerName="istio-proxy" Oct 30 18:33:59.316854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.316800 5232 cpu_manager.go:414] "containerMap: removing container" podUID="c5711415-04ea-4954-a672-2af1986aec4c" containerName="istio-proxy" Oct 30 18:33:59.316854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.316808 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5711415-04ea-4954-a672-2af1986aec4c" containerName="istio-proxy" Oct 30 18:33:59.316854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.316833 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7b1d7063-78cf-4a19-b2d0-e1f42d98dc73" containerName="revoke-keys" Oct 30 18:33:59.316854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.316841 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b1d7063-78cf-4a19-b2d0-e1f42d98dc73" containerName="revoke-keys" Oct 30 18:33:59.317024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.316857 5232 cpu_manager.go:414] "containerMap: removing container" podUID="3480923d-8625-430e-819c-628e203c4946" containerName="istio-proxy" Oct 30 18:33:59.317024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.316865 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3480923d-8625-430e-819c-628e203c4946" containerName="istio-proxy" Oct 30 18:33:59.317024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.316888 5232 cpu_manager.go:414] "containerMap: removing container" podUID="3480923d-8625-430e-819c-628e203c4946" containerName="istio-proxy" Oct 30 18:33:59.317024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.316894 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3480923d-8625-430e-819c-628e203c4946" containerName="istio-proxy" Oct 30 18:33:59.317024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.316992 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5711415-04ea-4954-a672-2af1986aec4c" containerName="istio-proxy" Oct 30 18:33:59.317024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.317009 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b1d7063-78cf-4a19-b2d0-e1f42d98dc73" containerName="revoke-keys" Oct 30 18:33:59.317024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.317019 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="3480923d-8625-430e-819c-628e203c4946" containerName="istio-proxy" Oct 30 18:33:59.317259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.317030 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="3480923d-8625-430e-819c-628e203c4946" containerName="istio-proxy" Oct 30 18:33:59.321553 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.321526 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt" Oct 30 18:33:59.336650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.336621 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt"] Oct 30 18:33:59.338283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.338232 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:33:59.338417 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.338336 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:33:59.361494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.361431 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf142398-5923-4658-bdf2-cd3c76b0ef93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a93233a0-kjrwt\" (UID: \"cf142398-5923-4658-bdf2-cd3c76b0ef93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt" Oct 30 18:33:59.361494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.361487 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kw5nw\" (UniqueName: \"kubernetes.io/projected/cf142398-5923-4658-bdf2-cd3c76b0ef93-kube-api-access-kw5nw\") pod \"maas-api-revoke-sub-a93233a0-kjrwt\" (UID: \"cf142398-5923-4658-bdf2-cd3c76b0ef93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt" Oct 30 18:33:59.361789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.361557 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf142398-5923-4658-bdf2-cd3c76b0ef93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a93233a0-kjrwt\" (UID: \"cf142398-5923-4658-bdf2-cd3c76b0ef93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt" Oct 30 18:33:59.462216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.462184 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kw5nw\" (UniqueName: \"kubernetes.io/projected/cf142398-5923-4658-bdf2-cd3c76b0ef93-kube-api-access-kw5nw\") pod \"maas-api-revoke-sub-a93233a0-kjrwt\" (UID: \"cf142398-5923-4658-bdf2-cd3c76b0ef93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt" Oct 30 18:33:59.462395 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.462228 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf142398-5923-4658-bdf2-cd3c76b0ef93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a93233a0-kjrwt\" (UID: \"cf142398-5923-4658-bdf2-cd3c76b0ef93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt" Oct 30 18:33:59.462395 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.462299 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf142398-5923-4658-bdf2-cd3c76b0ef93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a93233a0-kjrwt\" (UID: \"cf142398-5923-4658-bdf2-cd3c76b0ef93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt" Oct 30 18:33:59.462944 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.462910 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf142398-5923-4658-bdf2-cd3c76b0ef93-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a93233a0-kjrwt\" (UID: \"cf142398-5923-4658-bdf2-cd3c76b0ef93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt" Oct 30 18:33:59.464838 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.464818 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf142398-5923-4658-bdf2-cd3c76b0ef93-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a93233a0-kjrwt\" (UID: \"cf142398-5923-4658-bdf2-cd3c76b0ef93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt" Oct 30 18:33:59.480728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.480699 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kw5nw\" (UniqueName: \"kubernetes.io/projected/cf142398-5923-4658-bdf2-cd3c76b0ef93-kube-api-access-kw5nw\") pod \"maas-api-revoke-sub-a93233a0-kjrwt\" (UID: \"cf142398-5923-4658-bdf2-cd3c76b0ef93\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt" Oct 30 18:33:59.634695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.634619 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt" Oct 30 18:33:59.938354 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:33:59.938327 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt"] Oct 30 18:33:59.940101 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:33:59.940072 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf142398_5923_4658_bdf2_cd3c76b0ef93.slice/crio-a7fef319605feff7571e1a9761fc23b0fcf1ef789bb2df573c12507d57e2aab7 WatchSource:0}: Error finding container a7fef319605feff7571e1a9761fc23b0fcf1ef789bb2df573c12507d57e2aab7: Status 404 returned error can't find the container with id a7fef319605feff7571e1a9761fc23b0fcf1ef789bb2df573c12507d57e2aab7 Oct 30 18:34:00.027647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:00.027604 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc"] Oct 30 18:34:00.030473 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:00.030436 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d"] Oct 30 18:34:00.032190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:00.032171 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-75673761-hlxbc"] Oct 30 18:34:00.039317 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:00.039296 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-47820c98-v8h9d"] Oct 30 18:34:00.785659 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:00.785630 5232 generic.go:356] "Generic (PLEG): container finished" podID="cf142398-5923-4658-bdf2-cd3c76b0ef93" containerID="79a9432425ed220d73e1fcf831e5578090442da4f838505f3036ebb321e6c641" exitCode=0 Oct 30 18:34:00.786031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:00.785721 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt" event={"ID":"cf142398-5923-4658-bdf2-cd3c76b0ef93","Type":"ContainerDied","Data":"79a9432425ed220d73e1fcf831e5578090442da4f838505f3036ebb321e6c641"} Oct 30 18:34:00.786031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:00.785774 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt" event={"ID":"cf142398-5923-4658-bdf2-cd3c76b0ef93","Type":"ContainerStarted","Data":"a7fef319605feff7571e1a9761fc23b0fcf1ef789bb2df573c12507d57e2aab7"} Oct 30 18:34:00.801704 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:00.801680 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9b2c9fa-68ce-4152-ba7a-7440e1274eee" path="/var/lib/kubelet/pods/a9b2c9fa-68ce-4152-ba7a-7440e1274eee/volumes" Oct 30 18:34:00.802107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:00.802094 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49" path="/var/lib/kubelet/pods/e34d5fcc-8eaf-41b1-a11e-5c2ca5edcf49/volumes" Oct 30 18:34:01.924254 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:01.924228 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt" Oct 30 18:34:01.986344 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:01.986315 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cf142398-5923-4658-bdf2-cd3c76b0ef93-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf142398-5923-4658-bdf2-cd3c76b0ef93-maas-api-xks-ca\") pod \"cf142398-5923-4658-bdf2-cd3c76b0ef93\" (UID: \"cf142398-5923-4658-bdf2-cd3c76b0ef93\") " Oct 30 18:34:01.986539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:01.986405 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf142398-5923-4658-bdf2-cd3c76b0ef93-kube-api-access-kw5nw\" (UniqueName: \"kubernetes.io/projected/cf142398-5923-4658-bdf2-cd3c76b0ef93-kube-api-access-kw5nw\") pod \"cf142398-5923-4658-bdf2-cd3c76b0ef93\" (UID: \"cf142398-5923-4658-bdf2-cd3c76b0ef93\") " Oct 30 18:34:01.986539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:01.986445 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf142398-5923-4658-bdf2-cd3c76b0ef93-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf142398-5923-4658-bdf2-cd3c76b0ef93-maas-api-openshift-ca\") pod \"cf142398-5923-4658-bdf2-cd3c76b0ef93\" (UID: \"cf142398-5923-4658-bdf2-cd3c76b0ef93\") " Oct 30 18:34:01.986821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:01.986799 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf142398-5923-4658-bdf2-cd3c76b0ef93-maas-api-openshift-ca" pod "cf142398-5923-4658-bdf2-cd3c76b0ef93" (UID: "cf142398-5923-4658-bdf2-cd3c76b0ef93"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:01.988633 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:01.988607 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cf142398-5923-4658-bdf2-cd3c76b0ef93-maas-api-xks-ca" pod "cf142398-5923-4658-bdf2-cd3c76b0ef93" (UID: "cf142398-5923-4658-bdf2-cd3c76b0ef93"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:01.988732 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:01.988714 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf142398-5923-4658-bdf2-cd3c76b0ef93-kube-api-access-kw5nw" pod "cf142398-5923-4658-bdf2-cd3c76b0ef93" (UID: "cf142398-5923-4658-bdf2-cd3c76b0ef93"). InnerVolumeSpecName "kube-api-access-kw5nw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:02.087651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:02.087573 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cf142398-5923-4658-bdf2-cd3c76b0ef93-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:02.087651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:02.087600 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kw5nw\" (UniqueName: \"kubernetes.io/projected/cf142398-5923-4658-bdf2-cd3c76b0ef93-kube-api-access-kw5nw\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:02.087651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:02.087610 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cf142398-5923-4658-bdf2-cd3c76b0ef93-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:02.799840 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:02.799811 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt" Oct 30 18:34:02.802427 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:02.802402 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt" event={"ID":"cf142398-5923-4658-bdf2-cd3c76b0ef93","Type":"ContainerDied","Data":"a7fef319605feff7571e1a9761fc23b0fcf1ef789bb2df573c12507d57e2aab7"} Oct 30 18:34:02.802590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:02.802443 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7fef319605feff7571e1a9761fc23b0fcf1ef789bb2df573c12507d57e2aab7" Oct 30 18:34:04.239332 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:04.239275 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" podUID="7b743aee-2a49-44f2-aaae-24cd2e760827" containerName="payload-processing" containerID="cri-o://db648949b8f07000f2740bbaef0f28e67df628c8f38d478ed8f54e55698dfa19" gracePeriod=25 Oct 30 18:34:08.486914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.486869 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" containerID="cri-o://f13b7d4f36eea302662aaf102649aa9b6ac553af0fd2de194b7c46a770302a18" gracePeriod=45 Oct 30 18:34:08.515252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.515216 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" containerID="cri-o://6b6b52f44ba1e7d8d967263c7bb390e46b52144b0b070185539fd21f7db63271" gracePeriod=45 Oct 30 18:34:08.758241 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.758218 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:34:08.787126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.787092 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:34:08.831916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.831887 5232 generic.go:356] "Generic (PLEG): container finished" podID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerID="6b6b52f44ba1e7d8d967263c7bb390e46b52144b0b070185539fd21f7db63271" exitCode=0 Oct 30 18:34:08.832100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.831932 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" Oct 30 18:34:08.832100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.831981 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" event={"ID":"52075caf-2e4a-4c47-9a7a-9a9fcc744277","Type":"ContainerDied","Data":"6b6b52f44ba1e7d8d967263c7bb390e46b52144b0b070185539fd21f7db63271"} Oct 30 18:34:08.832100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.832027 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2" event={"ID":"52075caf-2e4a-4c47-9a7a-9a9fcc744277","Type":"ContainerDied","Data":"3409526554e35b29a5afc0e90221008a2614cdcea1a8d16325be3d8a08246af0"} Oct 30 18:34:08.832100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.832049 5232 scope.go:122] "RemoveContainer" containerID="6b6b52f44ba1e7d8d967263c7bb390e46b52144b0b070185539fd21f7db63271" Oct 30 18:34:08.833388 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.833365 5232 generic.go:356] "Generic (PLEG): container finished" podID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerID="f13b7d4f36eea302662aaf102649aa9b6ac553af0fd2de194b7c46a770302a18" exitCode=0 Oct 30 18:34:08.833546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.833448 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" event={"ID":"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f","Type":"ContainerDied","Data":"f13b7d4f36eea302662aaf102649aa9b6ac553af0fd2de194b7c46a770302a18"} Oct 30 18:34:08.833546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.833477 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" event={"ID":"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f","Type":"ContainerDied","Data":"22ed88480cd30e03c95909459de5163ff3e197c183408bdbde03319654c51781"} Oct 30 18:34:08.833546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.833543 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm" Oct 30 18:34:08.847765 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.847742 5232 scope.go:122] "RemoveContainer" containerID="6b6b52f44ba1e7d8d967263c7bb390e46b52144b0b070185539fd21f7db63271" Oct 30 18:34:08.848029 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.848006 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-home\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-home\") pod \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " Oct 30 18:34:08.848607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.848058 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-dshm\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-dshm\") pod \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " Oct 30 18:34:08.848607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.848090 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-home\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-home\") pod \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " Oct 30 18:34:08.848607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.848128 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-kube-api-access-q4cbt\" (UniqueName: \"kubernetes.io/projected/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-kube-api-access-q4cbt\") pod \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " Oct 30 18:34:08.848607 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:34:08.848143 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6b6b52f44ba1e7d8d967263c7bb390e46b52144b0b070185539fd21f7db63271\": container with ID starting with 6b6b52f44ba1e7d8d967263c7bb390e46b52144b0b070185539fd21f7db63271 not found: ID does not exist" containerID="6b6b52f44ba1e7d8d967263c7bb390e46b52144b0b070185539fd21f7db63271" Oct 30 18:34:08.848607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.848164 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-dshm\") pod \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " Oct 30 18:34:08.848607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.848184 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6b6b52f44ba1e7d8d967263c7bb390e46b52144b0b070185539fd21f7db63271"} err="failed to get container status \"6b6b52f44ba1e7d8d967263c7bb390e46b52144b0b070185539fd21f7db63271\": rpc error: code = NotFound desc = could not find container \"6b6b52f44ba1e7d8d967263c7bb390e46b52144b0b070185539fd21f7db63271\": container with ID starting with 6b6b52f44ba1e7d8d967263c7bb390e46b52144b0b070185539fd21f7db63271 not found: ID does not exist" Oct 30 18:34:08.848607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.848211 5232 scope.go:122] "RemoveContainer" containerID="f13b7d4f36eea302662aaf102649aa9b6ac553af0fd2de194b7c46a770302a18" Oct 30 18:34:08.848607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.848195 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-model-cache\") pod \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " Oct 30 18:34:08.848607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.848359 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/52075caf-2e4a-4c47-9a7a-9a9fcc744277-tls-certs\" (UniqueName: \"kubernetes.io/secret/52075caf-2e4a-4c47-9a7a-9a9fcc744277-tls-certs\") pod \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " Oct 30 18:34:08.848607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.848402 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-model-cache" pod "ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" (UID: "ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:08.848607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.848429 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-tls-certs\") pod \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " Oct 30 18:34:08.848607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.848479 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52075caf-2e4a-4c47-9a7a-9a9fcc744277-kube-api-access-x6x58\" (UniqueName: \"kubernetes.io/projected/52075caf-2e4a-4c47-9a7a-9a9fcc744277-kube-api-access-x6x58\") pod \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " Oct 30 18:34:08.848607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.848547 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-tmp-dir\") pod \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " Oct 30 18:34:08.848607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.848594 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-tmp-dir\") pod \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\" (UID: \"ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f\") " Oct 30 18:34:08.849465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.848644 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-model-cache\") pod \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\" (UID: \"52075caf-2e4a-4c47-9a7a-9a9fcc744277\") " Oct 30 18:34:08.849465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.849006 5232 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-model-cache\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:08.849465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.849009 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-home" pod "52075caf-2e4a-4c47-9a7a-9a9fcc744277" (UID: "52075caf-2e4a-4c47-9a7a-9a9fcc744277"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:08.849465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.849120 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-home" pod "ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" (UID: "ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:08.849465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.849296 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-tmp-dir" pod "ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" (UID: "ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:08.849465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.849318 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-tmp-dir" pod "52075caf-2e4a-4c47-9a7a-9a9fcc744277" (UID: "52075caf-2e4a-4c47-9a7a-9a9fcc744277"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:08.850023 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.849477 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-model-cache" pod "52075caf-2e4a-4c47-9a7a-9a9fcc744277" (UID: "52075caf-2e4a-4c47-9a7a-9a9fcc744277"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:08.853731 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.853696 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/52075caf-2e4a-4c47-9a7a-9a9fcc744277-tls-certs" pod "52075caf-2e4a-4c47-9a7a-9a9fcc744277" (UID: "52075caf-2e4a-4c47-9a7a-9a9fcc744277"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:08.853873 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.853846 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-tls-certs" pod "ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" (UID: "ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:08.853991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.853870 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-dshm" pod "ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" (UID: "ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:08.853991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.853920 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52075caf-2e4a-4c47-9a7a-9a9fcc744277-kube-api-access-x6x58" pod "52075caf-2e4a-4c47-9a7a-9a9fcc744277" (UID: "52075caf-2e4a-4c47-9a7a-9a9fcc744277"). InnerVolumeSpecName "kube-api-access-x6x58". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:08.853991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.853935 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-dshm" pod "52075caf-2e4a-4c47-9a7a-9a9fcc744277" (UID: "52075caf-2e4a-4c47-9a7a-9a9fcc744277"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:08.854182 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.854161 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-kube-api-access-q4cbt" pod "ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" (UID: "ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f"). InnerVolumeSpecName "kube-api-access-q4cbt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:08.862743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.862719 5232 scope.go:122] "RemoveContainer" containerID="f13b7d4f36eea302662aaf102649aa9b6ac553af0fd2de194b7c46a770302a18" Oct 30 18:34:08.863130 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:34:08.863108 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f13b7d4f36eea302662aaf102649aa9b6ac553af0fd2de194b7c46a770302a18\": container with ID starting with f13b7d4f36eea302662aaf102649aa9b6ac553af0fd2de194b7c46a770302a18 not found: ID does not exist" containerID="f13b7d4f36eea302662aaf102649aa9b6ac553af0fd2de194b7c46a770302a18" Oct 30 18:34:08.863200 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.863140 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f13b7d4f36eea302662aaf102649aa9b6ac553af0fd2de194b7c46a770302a18"} err="failed to get container status \"f13b7d4f36eea302662aaf102649aa9b6ac553af0fd2de194b7c46a770302a18\": rpc error: code = NotFound desc = could not find container \"f13b7d4f36eea302662aaf102649aa9b6ac553af0fd2de194b7c46a770302a18\": container with ID starting with f13b7d4f36eea302662aaf102649aa9b6ac553af0fd2de194b7c46a770302a18 not found: ID does not exist" Oct 30 18:34:08.950138 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.950099 5232 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-model-cache\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:08.950138 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.950137 5232 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-home\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:08.950365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.950149 5232 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-dshm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:08.950365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.950161 5232 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-home\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:08.950365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.950176 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q4cbt\" (UniqueName: \"kubernetes.io/projected/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-kube-api-access-q4cbt\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:08.950365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.950188 5232 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-dshm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:08.950365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.950217 5232 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/52075caf-2e4a-4c47-9a7a-9a9fcc744277-tls-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:08.950365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.950232 5232 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-tls-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:08.950365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.950243 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6x58\" (UniqueName: \"kubernetes.io/projected/52075caf-2e4a-4c47-9a7a-9a9fcc744277-kube-api-access-x6x58\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:08.950365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.950255 5232 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/52075caf-2e4a-4c47-9a7a-9a9fcc744277-tmp-dir\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:08.950365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:08.950270 5232 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f-tmp-dir\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:09.176487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:09.176458 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2"] Oct 30 18:34:09.179222 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:09.179199 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/simulator-w4-4ef6ac-kserve-5f7b8d8fbb-stbx2"] Oct 30 18:34:09.221646 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:09.221617 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm"] Oct 30 18:34:09.225245 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:09.225225 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-4ef6ac/distinct-w4-4ef6ac-kserve-79696c78c8-6g6zm"] Oct 30 18:34:10.800823 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:10.800784 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" path="/var/lib/kubelet/pods/52075caf-2e4a-4c47-9a7a-9a9fcc744277/volumes" Oct 30 18:34:10.801215 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:10.801181 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" path="/var/lib/kubelet/pods/ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f/volumes" Oct 30 18:34:11.021579 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:11.021542 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" podUID="7b743aee-2a49-44f2-aaae-24cd2e760827" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:34:15.108643 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:15.108609 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c"] Oct 30 18:34:15.130522 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:15.121166 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-4ef6ac-jqf5c"] Oct 30 18:34:16.801019 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:16.800982 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b1d7063-78cf-4a19-b2d0-e1f42d98dc73" path="/var/lib/kubelet/pods/7b1d7063-78cf-4a19-b2d0-e1f42d98dc73/volumes" Oct 30 18:34:21.021821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:21.021784 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" podUID="7b743aee-2a49-44f2-aaae-24cd2e760827" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:34:24.485048 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:24.485017 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" Oct 30 18:34:24.593084 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:24.593000 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b743aee-2a49-44f2-aaae-24cd2e760827-kube-api-access-m9p2j\" (UniqueName: \"kubernetes.io/projected/7b743aee-2a49-44f2-aaae-24cd2e760827-kube-api-access-m9p2j\") pod \"7b743aee-2a49-44f2-aaae-24cd2e760827\" (UID: \"7b743aee-2a49-44f2-aaae-24cd2e760827\") " Oct 30 18:34:24.593274 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:24.593133 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b743aee-2a49-44f2-aaae-24cd2e760827-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b743aee-2a49-44f2-aaae-24cd2e760827-plugins-config-volume\") pod \"7b743aee-2a49-44f2-aaae-24cd2e760827\" (UID: \"7b743aee-2a49-44f2-aaae-24cd2e760827\") " Oct 30 18:34:24.593591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:24.593563 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b743aee-2a49-44f2-aaae-24cd2e760827-plugins-config-volume" pod "7b743aee-2a49-44f2-aaae-24cd2e760827" (UID: "7b743aee-2a49-44f2-aaae-24cd2e760827"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:24.595407 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:24.595382 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b743aee-2a49-44f2-aaae-24cd2e760827-kube-api-access-m9p2j" pod "7b743aee-2a49-44f2-aaae-24cd2e760827" (UID: "7b743aee-2a49-44f2-aaae-24cd2e760827"). InnerVolumeSpecName "kube-api-access-m9p2j". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:24.693974 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:24.693944 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9p2j\" (UniqueName: \"kubernetes.io/projected/7b743aee-2a49-44f2-aaae-24cd2e760827-kube-api-access-m9p2j\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:24.693974 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:24.693971 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b743aee-2a49-44f2-aaae-24cd2e760827-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:24.937595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:24.937492 5232 generic.go:356] "Generic (PLEG): container finished" podID="7b743aee-2a49-44f2-aaae-24cd2e760827" containerID="db648949b8f07000f2740bbaef0f28e67df628c8f38d478ed8f54e55698dfa19" exitCode=0 Oct 30 18:34:24.937595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:24.937538 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" event={"ID":"7b743aee-2a49-44f2-aaae-24cd2e760827","Type":"ContainerDied","Data":"db648949b8f07000f2740bbaef0f28e67df628c8f38d478ed8f54e55698dfa19"} Oct 30 18:34:24.937595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:24.937583 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" Oct 30 18:34:24.937818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:24.937605 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw" event={"ID":"7b743aee-2a49-44f2-aaae-24cd2e760827","Type":"ContainerDied","Data":"adbf7bb0994cec247cacf7e0cea5a3f3de2270a12327541b1bc20086d40226c6"} Oct 30 18:34:24.937818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:24.937632 5232 scope.go:122] "RemoveContainer" containerID="db648949b8f07000f2740bbaef0f28e67df628c8f38d478ed8f54e55698dfa19" Oct 30 18:34:24.946992 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:24.946965 5232 scope.go:122] "RemoveContainer" containerID="db648949b8f07000f2740bbaef0f28e67df628c8f38d478ed8f54e55698dfa19" Oct 30 18:34:24.947289 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:34:24.947263 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"db648949b8f07000f2740bbaef0f28e67df628c8f38d478ed8f54e55698dfa19\": container with ID starting with db648949b8f07000f2740bbaef0f28e67df628c8f38d478ed8f54e55698dfa19 not found: ID does not exist" containerID="db648949b8f07000f2740bbaef0f28e67df628c8f38d478ed8f54e55698dfa19" Oct 30 18:34:24.947391 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:24.947295 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"db648949b8f07000f2740bbaef0f28e67df628c8f38d478ed8f54e55698dfa19"} err="failed to get container status \"db648949b8f07000f2740bbaef0f28e67df628c8f38d478ed8f54e55698dfa19\": rpc error: code = NotFound desc = could not find container \"db648949b8f07000f2740bbaef0f28e67df628c8f38d478ed8f54e55698dfa19\": container with ID starting with db648949b8f07000f2740bbaef0f28e67df628c8f38d478ed8f54e55698dfa19 not found: ID does not exist" Oct 30 18:34:24.987991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:24.987966 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw"] Oct 30 18:34:25.000309 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:25.000284 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-4ef6ac-cbfff8976-ldbrw"] Oct 30 18:34:26.800838 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:26.800802 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b743aee-2a49-44f2-aaae-24cd2e760827" path="/var/lib/kubelet/pods/7b743aee-2a49-44f2-aaae-24cd2e760827/volumes" Oct 30 18:34:27.032949 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:34:27.032912 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:34:27.718904 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:27.718873 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c97f9884-dsjzk"] Oct 30 18:34:27.719096 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:27.719060 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-68c97f9884-dsjzk" podUID="f76e41be-f30a-4c44-90cc-ef0b08127828" containerName="manager" containerID="cri-o://6d40f668e435acf97f6a97b17e6f88669431cd68c6ee6eb5c0ac74c6de85029f" gracePeriod=30 Oct 30 18:34:27.957792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:27.957764 5232 generic.go:356] "Generic (PLEG): container finished" podID="f76e41be-f30a-4c44-90cc-ef0b08127828" containerID="6d40f668e435acf97f6a97b17e6f88669431cd68c6ee6eb5c0ac74c6de85029f" exitCode=0 Oct 30 18:34:27.958212 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:27.957825 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c97f9884-dsjzk" event={"ID":"f76e41be-f30a-4c44-90cc-ef0b08127828","Type":"ContainerDied","Data":"6d40f668e435acf97f6a97b17e6f88669431cd68c6ee6eb5c0ac74c6de85029f"} Oct 30 18:34:27.958212 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:27.957881 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c97f9884-dsjzk" event={"ID":"f76e41be-f30a-4c44-90cc-ef0b08127828","Type":"ContainerDied","Data":"adb0d938d6816669157a767739395a21319d7ac9e3afeffc792c58d1369253c5"} Oct 30 18:34:27.958212 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:27.957902 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="adb0d938d6816669157a767739395a21319d7ac9e3afeffc792c58d1369253c5" Oct 30 18:34:27.958212 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:27.957915 5232 scope.go:122] "RemoveContainer" containerID="d5d32e7f2f8775dcbbc28f3ca718e746c9e85675da59073fd3b28667b8b0b5e8" Oct 30 18:34:27.967835 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:27.967811 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c97f9884-dsjzk" Oct 30 18:34:28.126632 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:28.126598 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f76e41be-f30a-4c44-90cc-ef0b08127828-webhook-cert\" (UniqueName: \"kubernetes.io/secret/f76e41be-f30a-4c44-90cc-ef0b08127828-webhook-cert\") pod \"f76e41be-f30a-4c44-90cc-ef0b08127828\" (UID: \"f76e41be-f30a-4c44-90cc-ef0b08127828\") " Oct 30 18:34:28.126632 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:28.126633 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f76e41be-f30a-4c44-90cc-ef0b08127828-metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76e41be-f30a-4c44-90cc-ef0b08127828-metrics-certs\") pod \"f76e41be-f30a-4c44-90cc-ef0b08127828\" (UID: \"f76e41be-f30a-4c44-90cc-ef0b08127828\") " Oct 30 18:34:28.126917 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:28.126671 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f76e41be-f30a-4c44-90cc-ef0b08127828-kube-api-access-jdcd7\" (UniqueName: \"kubernetes.io/projected/f76e41be-f30a-4c44-90cc-ef0b08127828-kube-api-access-jdcd7\") pod \"f76e41be-f30a-4c44-90cc-ef0b08127828\" (UID: \"f76e41be-f30a-4c44-90cc-ef0b08127828\") " Oct 30 18:34:28.128939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:28.128912 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76e41be-f30a-4c44-90cc-ef0b08127828-metrics-certs" pod "f76e41be-f30a-4c44-90cc-ef0b08127828" (UID: "f76e41be-f30a-4c44-90cc-ef0b08127828"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:28.129077 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:28.128977 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f76e41be-f30a-4c44-90cc-ef0b08127828-kube-api-access-jdcd7" pod "f76e41be-f30a-4c44-90cc-ef0b08127828" (UID: "f76e41be-f30a-4c44-90cc-ef0b08127828"). InnerVolumeSpecName "kube-api-access-jdcd7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:28.129077 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:28.128987 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f76e41be-f30a-4c44-90cc-ef0b08127828-webhook-cert" pod "f76e41be-f30a-4c44-90cc-ef0b08127828" (UID: "f76e41be-f30a-4c44-90cc-ef0b08127828"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:28.227976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:28.227935 5232 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/f76e41be-f30a-4c44-90cc-ef0b08127828-webhook-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.227976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:28.227967 5232 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/f76e41be-f30a-4c44-90cc-ef0b08127828-metrics-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.227976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:28.227977 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jdcd7\" (UniqueName: \"kubernetes.io/projected/f76e41be-f30a-4c44-90cc-ef0b08127828-kube-api-access-jdcd7\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:28.964845 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:28.964812 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c97f9884-dsjzk" Oct 30 18:34:29.041433 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.041401 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c97f9884-dsjzk"] Oct 30 18:34:29.043759 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.043725 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68c97f9884-dsjzk"] Oct 30 18:34:29.747682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.747651 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68c97f9884-qdl7l"] Oct 30 18:34:29.748065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748048 5232 cpu_manager.go:414] "containerMap: removing container" podUID="cf142398-5923-4658-bdf2-cd3c76b0ef93" containerName="revoke-keys" Oct 30 18:34:29.748162 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748068 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf142398-5923-4658-bdf2-cd3c76b0ef93" containerName="revoke-keys" Oct 30 18:34:29.748162 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748083 5232 cpu_manager.go:414] "containerMap: removing container" podUID="f76e41be-f30a-4c44-90cc-ef0b08127828" containerName="manager" Oct 30 18:34:29.748162 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748089 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76e41be-f30a-4c44-90cc-ef0b08127828" containerName="manager" Oct 30 18:34:29.748162 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748102 5232 cpu_manager.go:414] "containerMap: removing container" podUID="f76e41be-f30a-4c44-90cc-ef0b08127828" containerName="manager" Oct 30 18:34:29.748162 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748108 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f76e41be-f30a-4c44-90cc-ef0b08127828" containerName="manager" Oct 30 18:34:29.748162 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748138 5232 cpu_manager.go:414] "containerMap: removing container" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" Oct 30 18:34:29.748162 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748146 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" Oct 30 18:34:29.748596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748174 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7b743aee-2a49-44f2-aaae-24cd2e760827" containerName="payload-processing" Oct 30 18:34:29.748596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748182 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b743aee-2a49-44f2-aaae-24cd2e760827" containerName="payload-processing" Oct 30 18:34:29.748596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748193 5232 cpu_manager.go:414] "containerMap: removing container" podUID="c5711415-04ea-4954-a672-2af1986aec4c" containerName="istio-proxy" Oct 30 18:34:29.748596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748200 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5711415-04ea-4954-a672-2af1986aec4c" containerName="istio-proxy" Oct 30 18:34:29.748596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748216 5232 cpu_manager.go:414] "containerMap: removing container" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" Oct 30 18:34:29.748596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748223 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" Oct 30 18:34:29.748596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748302 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b743aee-2a49-44f2-aaae-24cd2e760827" containerName="payload-processing" Oct 30 18:34:29.748596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748317 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5711415-04ea-4954-a672-2af1986aec4c" containerName="istio-proxy" Oct 30 18:34:29.748596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748327 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="52075caf-2e4a-4c47-9a7a-9a9fcc744277" containerName="main" Oct 30 18:34:29.748596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748337 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf142398-5923-4658-bdf2-cd3c76b0ef93" containerName="revoke-keys" Oct 30 18:34:29.748596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748346 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76e41be-f30a-4c44-90cc-ef0b08127828" containerName="manager" Oct 30 18:34:29.748596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748355 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="f76e41be-f30a-4c44-90cc-ef0b08127828" containerName="manager" Oct 30 18:34:29.748596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.748367 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed2e6b15-13d5-4fa8-909b-42ecf5f35c1f" containerName="main" Oct 30 18:34:29.753033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.753014 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c97f9884-qdl7l" Oct 30 18:34:29.780372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.780337 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 18:34:29.780613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.780350 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 18:34:29.780779 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.780459 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 18:34:29.783427 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.783388 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c97f9884-qdl7l"] Oct 30 18:34:29.789261 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.789238 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 18:34:29.789519 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.789487 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-r2cqt\"" Oct 30 18:34:29.790146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.790093 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 18:34:29.842566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.842499 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-webhook-cert\") pod \"maas-controller-68c97f9884-qdl7l\" (UID: \"bb4dfa63-93b7-4a41-bf1b-1b9151b60655\") " pod="opendatahub/maas-controller-68c97f9884-qdl7l" Oct 30 18:34:29.842781 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.842635 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4hmp\" (UniqueName: \"kubernetes.io/projected/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-kube-api-access-d4hmp\") pod \"maas-controller-68c97f9884-qdl7l\" (UID: \"bb4dfa63-93b7-4a41-bf1b-1b9151b60655\") " pod="opendatahub/maas-controller-68c97f9884-qdl7l" Oct 30 18:34:29.842781 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.842745 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-metrics-certs\") pod \"maas-controller-68c97f9884-qdl7l\" (UID: \"bb4dfa63-93b7-4a41-bf1b-1b9151b60655\") " pod="opendatahub/maas-controller-68c97f9884-qdl7l" Oct 30 18:34:29.943988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.943955 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-metrics-certs\") pod \"maas-controller-68c97f9884-qdl7l\" (UID: \"bb4dfa63-93b7-4a41-bf1b-1b9151b60655\") " pod="opendatahub/maas-controller-68c97f9884-qdl7l" Oct 30 18:34:29.944192 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.944004 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-webhook-cert\") pod \"maas-controller-68c97f9884-qdl7l\" (UID: \"bb4dfa63-93b7-4a41-bf1b-1b9151b60655\") " pod="opendatahub/maas-controller-68c97f9884-qdl7l" Oct 30 18:34:29.944192 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.944067 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4hmp\" (UniqueName: \"kubernetes.io/projected/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-kube-api-access-d4hmp\") pod \"maas-controller-68c97f9884-qdl7l\" (UID: \"bb4dfa63-93b7-4a41-bf1b-1b9151b60655\") " pod="opendatahub/maas-controller-68c97f9884-qdl7l" Oct 30 18:34:29.946919 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.946889 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-metrics-certs\") pod \"maas-controller-68c97f9884-qdl7l\" (UID: \"bb4dfa63-93b7-4a41-bf1b-1b9151b60655\") " pod="opendatahub/maas-controller-68c97f9884-qdl7l" Oct 30 18:34:29.948448 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.948420 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-webhook-cert\") pod \"maas-controller-68c97f9884-qdl7l\" (UID: \"bb4dfa63-93b7-4a41-bf1b-1b9151b60655\") " pod="opendatahub/maas-controller-68c97f9884-qdl7l" Oct 30 18:34:29.963389 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:29.963360 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4hmp\" (UniqueName: \"kubernetes.io/projected/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-kube-api-access-d4hmp\") pod \"maas-controller-68c97f9884-qdl7l\" (UID: \"bb4dfa63-93b7-4a41-bf1b-1b9151b60655\") " pod="opendatahub/maas-controller-68c97f9884-qdl7l" Oct 30 18:34:30.070179 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:30.070104 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c97f9884-qdl7l" Oct 30 18:34:30.432585 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:30.429953 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c97f9884-qdl7l"] Oct 30 18:34:30.467551 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:30.465771 5232 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:34:30.801535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:30.801477 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f76e41be-f30a-4c44-90cc-ef0b08127828" path="/var/lib/kubelet/pods/f76e41be-f30a-4c44-90cc-ef0b08127828/volumes" Oct 30 18:34:31.003364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:31.003337 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c97f9884-qdl7l" event={"ID":"bb4dfa63-93b7-4a41-bf1b-1b9151b60655","Type":"ContainerStarted","Data":"4cbaf3779b6f2d32f87597b4b85560434034bf43ca55a974698fcae88ab001f3"} Oct 30 18:34:32.010189 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:32.010150 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c97f9884-qdl7l" event={"ID":"bb4dfa63-93b7-4a41-bf1b-1b9151b60655","Type":"ContainerStarted","Data":"a10dacdced632b5772fb63dea1e1fe1c1346b86fd772d1caec4c1ecf7e55a7e7"} Oct 30 18:34:32.010645 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:32.010218 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68c97f9884-qdl7l" Oct 30 18:34:32.051980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:32.051930 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-68c97f9884-qdl7l" podStartSLOduration=2.560255768 podStartE2EDuration="3.051917383s" totalImagesPullingTime="491.661615ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:32.050054516 +0000 UTC m=+1971.366474841" watchObservedRunningTime="2026-10-30 18:34:32.051917383 +0000 UTC m=+1971.368337704" Oct 30 18:34:33.276379 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:33.276317 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt"] Oct 30 18:34:33.276827 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:33.276553 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" podUID="0ea67986-73d8-4a9d-9268-a664b61c099c" containerName="istio-proxy" containerID="cri-o://dbe62416218fcc727e819bf7f386840897dcb415893c5624a6bac7a2704bfc5a" gracePeriod=30 Oct 30 18:34:37.430938 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:37.430902 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-cxtkt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:34:37.431329 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:37.430958 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" podUID="0ea67986-73d8-4a9d-9268-a664b61c099c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:34:38.519880 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.519854 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:34:38.556027 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.555964 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-token\" (UniqueName: \"kubernetes.io/projected/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-token\") pod \"0ea67986-73d8-4a9d-9268-a664b61c099c\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " Oct 30 18:34:38.556027 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.556005 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-podinfo\") pod \"0ea67986-73d8-4a9d-9268-a664b61c099c\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " Oct 30 18:34:38.556233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.556074 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-workload-certs\") pod \"0ea67986-73d8-4a9d-9268-a664b61c099c\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " Oct 30 18:34:38.556233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.556097 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0ea67986-73d8-4a9d-9268-a664b61c099c-kube-api-access-jg6p7\" (UniqueName: \"kubernetes.io/projected/0ea67986-73d8-4a9d-9268-a664b61c099c-kube-api-access-jg6p7\") pod \"0ea67986-73d8-4a9d-9268-a664b61c099c\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " Oct 30 18:34:38.556233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.556119 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-workload-socket\") pod \"0ea67986-73d8-4a9d-9268-a664b61c099c\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " Oct 30 18:34:38.556233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.556175 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0ea67986-73d8-4a9d-9268-a664b61c099c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea67986-73d8-4a9d-9268-a664b61c099c-istiod-ca-cert\") pod \"0ea67986-73d8-4a9d-9268-a664b61c099c\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " Oct 30 18:34:38.556233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.556216 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-credential-socket\") pod \"0ea67986-73d8-4a9d-9268-a664b61c099c\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " Oct 30 18:34:38.556542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.556247 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-data\") pod \"0ea67986-73d8-4a9d-9268-a664b61c099c\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " Oct 30 18:34:38.556542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.556306 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-envoy\") pod \"0ea67986-73d8-4a9d-9268-a664b61c099c\" (UID: \"0ea67986-73d8-4a9d-9268-a664b61c099c\") " Oct 30 18:34:38.556542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.556411 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-workload-certs" pod "0ea67986-73d8-4a9d-9268-a664b61c099c" (UID: "0ea67986-73d8-4a9d-9268-a664b61c099c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:38.556725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.556601 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-credential-socket" pod "0ea67986-73d8-4a9d-9268-a664b61c099c" (UID: "0ea67986-73d8-4a9d-9268-a664b61c099c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:38.556725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.556638 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:38.557007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.556802 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-workload-socket" pod "0ea67986-73d8-4a9d-9268-a664b61c099c" (UID: "0ea67986-73d8-4a9d-9268-a664b61c099c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:38.557007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.556915 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-data" pod "0ea67986-73d8-4a9d-9268-a664b61c099c" (UID: "0ea67986-73d8-4a9d-9268-a664b61c099c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:38.557007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.556946 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea67986-73d8-4a9d-9268-a664b61c099c-istiod-ca-cert" pod "0ea67986-73d8-4a9d-9268-a664b61c099c" (UID: "0ea67986-73d8-4a9d-9268-a664b61c099c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:38.558581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.558547 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-token" pod "0ea67986-73d8-4a9d-9268-a664b61c099c" (UID: "0ea67986-73d8-4a9d-9268-a664b61c099c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:38.558718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.558653 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-podinfo" pod "0ea67986-73d8-4a9d-9268-a664b61c099c" (UID: "0ea67986-73d8-4a9d-9268-a664b61c099c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:34:38.559008 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.558989 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ea67986-73d8-4a9d-9268-a664b61c099c-kube-api-access-jg6p7" pod "0ea67986-73d8-4a9d-9268-a664b61c099c" (UID: "0ea67986-73d8-4a9d-9268-a664b61c099c"). InnerVolumeSpecName "kube-api-access-jg6p7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:38.559076 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.559000 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-envoy" pod "0ea67986-73d8-4a9d-9268-a664b61c099c" (UID: "0ea67986-73d8-4a9d-9268-a664b61c099c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:38.657208 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.657179 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jg6p7\" (UniqueName: \"kubernetes.io/projected/0ea67986-73d8-4a9d-9268-a664b61c099c-kube-api-access-jg6p7\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:38.657208 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.657206 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:38.657208 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.657217 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0ea67986-73d8-4a9d-9268-a664b61c099c-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:38.657445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.657226 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:38.657445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.657234 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:38.657445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.657242 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:38.657445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.657249 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:38.657445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:38.657256 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0ea67986-73d8-4a9d-9268-a664b61c099c-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:39.047103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:39.047071 5232 generic.go:356] "Generic (PLEG): container finished" podID="0ea67986-73d8-4a9d-9268-a664b61c099c" containerID="dbe62416218fcc727e819bf7f386840897dcb415893c5624a6bac7a2704bfc5a" exitCode=0 Oct 30 18:34:39.047288 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:39.047125 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" Oct 30 18:34:39.047288 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:39.047161 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" event={"ID":"0ea67986-73d8-4a9d-9268-a664b61c099c","Type":"ContainerDied","Data":"dbe62416218fcc727e819bf7f386840897dcb415893c5624a6bac7a2704bfc5a"} Oct 30 18:34:39.047288 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:39.047223 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt" event={"ID":"0ea67986-73d8-4a9d-9268-a664b61c099c","Type":"ContainerDied","Data":"0235ea435e7c95f5e8ffe39b21ced50e0363cf7d6f5ef4f3fba5da2a0a70bab0"} Oct 30 18:34:39.047288 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:39.047251 5232 scope.go:122] "RemoveContainer" containerID="dbe62416218fcc727e819bf7f386840897dcb415893c5624a6bac7a2704bfc5a" Oct 30 18:34:39.056655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:39.056631 5232 scope.go:122] "RemoveContainer" containerID="dbe62416218fcc727e819bf7f386840897dcb415893c5624a6bac7a2704bfc5a" Oct 30 18:34:39.056909 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:34:39.056894 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dbe62416218fcc727e819bf7f386840897dcb415893c5624a6bac7a2704bfc5a\": container with ID starting with dbe62416218fcc727e819bf7f386840897dcb415893c5624a6bac7a2704bfc5a not found: ID does not exist" containerID="dbe62416218fcc727e819bf7f386840897dcb415893c5624a6bac7a2704bfc5a" Oct 30 18:34:39.056962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:39.056920 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dbe62416218fcc727e819bf7f386840897dcb415893c5624a6bac7a2704bfc5a"} err="failed to get container status \"dbe62416218fcc727e819bf7f386840897dcb415893c5624a6bac7a2704bfc5a\": rpc error: code = NotFound desc = could not find container \"dbe62416218fcc727e819bf7f386840897dcb415893c5624a6bac7a2704bfc5a\": container with ID starting with dbe62416218fcc727e819bf7f386840897dcb415893c5624a6bac7a2704bfc5a not found: ID does not exist" Oct 30 18:34:39.112357 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:39.112286 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt"] Oct 30 18:34:39.124776 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:39.124749 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-cxtkt"] Oct 30 18:34:40.801847 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:40.801813 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0ea67986-73d8-4a9d-9268-a664b61c099c" path="/var/lib/kubelet/pods/0ea67986-73d8-4a9d-9268-a664b61c099c/volumes" Oct 30 18:34:43.018272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:43.018246 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68c97f9884-qdl7l" Oct 30 18:34:46.025195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:46.025164 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj"] Oct 30 18:34:46.037260 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:46.037232 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs"] Oct 30 18:34:46.042383 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:46.042343 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-01d67609-vl9cj"] Oct 30 18:34:46.042549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:46.042398 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-531d7f2f-675vs"] Oct 30 18:34:46.799997 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:46.799964 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="476a0b93-3e62-4ae7-8287-4f620e7e016b" path="/var/lib/kubelet/pods/476a0b93-3e62-4ae7-8287-4f620e7e016b/volumes" Oct 30 18:34:46.800368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:46.800355 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0" path="/var/lib/kubelet/pods/5e6476fd-f31d-4f29-a5c7-4e51df8ff9f0/volumes" Oct 30 18:34:48.349851 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.349814 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr"] Oct 30 18:34:48.350425 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.350172 5232 cpu_manager.go:414] "containerMap: removing container" podUID="0ea67986-73d8-4a9d-9268-a664b61c099c" containerName="istio-proxy" Oct 30 18:34:48.350425 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.350182 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0ea67986-73d8-4a9d-9268-a664b61c099c" containerName="istio-proxy" Oct 30 18:34:48.350425 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.350239 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="0ea67986-73d8-4a9d-9268-a664b61c099c" containerName="istio-proxy" Oct 30 18:34:48.354897 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.354875 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.362199 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.362172 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt"] Oct 30 18:34:48.366252 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.366228 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.374084 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.374059 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr"] Oct 30 18:34:48.393624 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.393597 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt"] Oct 30 18:34:48.443606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.443573 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92403b9c-a622-457e-8b9b-a92f86f93b62-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.443803 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.443614 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.443803 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.443645 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92403b9c-a622-457e-8b9b-a92f86f93b62-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.443803 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.443775 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.443943 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.443811 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.443943 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.443887 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f248n\" (UniqueName: \"kubernetes.io/projected/92403b9c-a622-457e-8b9b-a92f86f93b62-kube-api-access-f248n\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.443943 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.443924 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.444079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.443957 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.444079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.444018 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92403b9c-a622-457e-8b9b-a92f86f93b62-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.444079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.444043 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92403b9c-a622-457e-8b9b-a92f86f93b62-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.444079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.444059 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92403b9c-a622-457e-8b9b-a92f86f93b62-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.444249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.444111 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.444249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.444131 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92403b9c-a622-457e-8b9b-a92f86f93b62-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.444249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.444180 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92403b9c-a622-457e-8b9b-a92f86f93b62-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.444249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.444216 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92403b9c-a622-457e-8b9b-a92f86f93b62-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.444428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.444266 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9kvtd\" (UniqueName: \"kubernetes.io/projected/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-kube-api-access-9kvtd\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.444428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.444284 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.444428 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.444304 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.544781 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.544749 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.544781 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.544779 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.545003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.544799 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f248n\" (UniqueName: \"kubernetes.io/projected/92403b9c-a622-457e-8b9b-a92f86f93b62-kube-api-access-f248n\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.545003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.544820 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.545003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.544843 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.545003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.544970 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92403b9c-a622-457e-8b9b-a92f86f93b62-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.545234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.545017 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92403b9c-a622-457e-8b9b-a92f86f93b62-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.545234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.545048 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92403b9c-a622-457e-8b9b-a92f86f93b62-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.545234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.545075 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.545234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.545099 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92403b9c-a622-457e-8b9b-a92f86f93b62-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.545234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.545125 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92403b9c-a622-457e-8b9b-a92f86f93b62-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.545234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.545161 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92403b9c-a622-457e-8b9b-a92f86f93b62-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.545234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.545198 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9kvtd\" (UniqueName: \"kubernetes.io/projected/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-kube-api-access-9kvtd\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.545234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.545206 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.545234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.545227 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.545769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.545255 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.545769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.545307 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92403b9c-a622-457e-8b9b-a92f86f93b62-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.545769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.545360 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.545769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.545390 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92403b9c-a622-457e-8b9b-a92f86f93b62-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.545769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.545405 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92403b9c-a622-457e-8b9b-a92f86f93b62-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.545769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.545446 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92403b9c-a622-457e-8b9b-a92f86f93b62-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.545769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.545467 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92403b9c-a622-457e-8b9b-a92f86f93b62-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.545769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.545689 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-workload-certs\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.545769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.545688 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92403b9c-a622-457e-8b9b-a92f86f93b62-istio-data\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.546454 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.546103 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-credential-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.546454 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.546168 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92403b9c-a622-457e-8b9b-a92f86f93b62-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.546454 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.546226 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.546454 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.546437 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-workload-socket\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.548060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.548035 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92403b9c-a622-457e-8b9b-a92f86f93b62-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.548294 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.548278 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-istio-envoy\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.548685 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.548665 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.548848 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.548826 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92403b9c-a622-457e-8b9b-a92f86f93b62-istio-podinfo\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.564403 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.564378 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.564521 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.564468 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f248n\" (UniqueName: \"kubernetes.io/projected/92403b9c-a622-457e-8b9b-a92f86f93b62-kube-api-access-f248n\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.564909 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.564893 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9kvtd\" (UniqueName: \"kubernetes.io/projected/b075648b-215b-4d0e-b3dc-41ea5ed46ac2-kube-api-access-9kvtd\") pod \"maas-default-gateway-openshift-default-77f556d647-zzlvr\" (UID: \"b075648b-215b-4d0e-b3dc-41ea5ed46ac2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.564978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.564961 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92403b9c-a622-457e-8b9b-a92f86f93b62-istio-token\") pod \"maas-default-gateway-openshift-default-77f556d647-kbsdt\" (UID: \"92403b9c-a622-457e-8b9b-a92f86f93b62\") " pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.668036 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.667965 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:48.681343 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.681314 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:48.985488 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.985460 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt"] Oct 30 18:34:48.987138 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:34:48.987110 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92403b9c_a622_457e_8b9b_a92f86f93b62.slice/crio-489c3870deb138719f67cd5d68a105bb7a9f319fdb7ca156a96ac76f311236fe WatchSource:0}: Error finding container 489c3870deb138719f67cd5d68a105bb7a9f319fdb7ca156a96ac76f311236fe: Status 404 returned error can't find the container with id 489c3870deb138719f67cd5d68a105bb7a9f319fdb7ca156a96ac76f311236fe Oct 30 18:34:48.988241 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.988209 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr"] Oct 30 18:34:48.989364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.989336 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:34:48.989423 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.989410 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:34:48.989460 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.989443 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:34:48.997786 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:34:48.997762 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb075648b_215b_4d0e_b3dc_41ea5ed46ac2.slice/crio-572fada05c402cfbf0722de6f44f0b8f16c8df835f641a72e8a2aef664d72227 WatchSource:0}: Error finding container 572fada05c402cfbf0722de6f44f0b8f16c8df835f641a72e8a2aef664d72227: Status 404 returned error can't find the container with id 572fada05c402cfbf0722de6f44f0b8f16c8df835f641a72e8a2aef664d72227 Oct 30 18:34:49.000009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:48.999981 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:34:49.000088 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.000039 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:34:49.000088 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.000067 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:34:49.106592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.106554 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" event={"ID":"b075648b-215b-4d0e-b3dc-41ea5ed46ac2","Type":"ContainerStarted","Data":"b81fe1085cbfa59dda31034c48f707eac3bd0a74489e8c77b166e89580dd371c"} Oct 30 18:34:49.106797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.106619 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" event={"ID":"b075648b-215b-4d0e-b3dc-41ea5ed46ac2","Type":"ContainerStarted","Data":"572fada05c402cfbf0722de6f44f0b8f16c8df835f641a72e8a2aef664d72227"} Oct 30 18:34:49.108289 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.108259 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" event={"ID":"92403b9c-a622-457e-8b9b-a92f86f93b62","Type":"ContainerStarted","Data":"e414c51fc738104ff33412ca8718aacb4ac7724210ad01aa74b9a2cbe77cb1eb"} Oct 30 18:34:49.108433 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.108303 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" event={"ID":"92403b9c-a622-457e-8b9b-a92f86f93b62","Type":"ContainerStarted","Data":"489c3870deb138719f67cd5d68a105bb7a9f319fdb7ca156a96ac76f311236fe"} Oct 30 18:34:49.150410 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.149603 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" podStartSLOduration=1.149583749 podStartE2EDuration="1.149583749s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:49.146106773 +0000 UTC m=+1988.462527095" watchObservedRunningTime="2026-10-30 18:34:49.149583749 +0000 UTC m=+1988.466004073" Oct 30 18:34:49.187063 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.187005 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" podStartSLOduration=1.186990198 podStartE2EDuration="1.186990198s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:34:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:34:49.185180224 +0000 UTC m=+1988.501600568" watchObservedRunningTime="2026-10-30 18:34:49.186990198 +0000 UTC m=+1988.503410518" Oct 30 18:34:49.276421 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.276382 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6"] Oct 30 18:34:49.281415 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.281390 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6" Oct 30 18:34:49.299788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.296556 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:34:49.299788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.296618 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:34:49.299788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.298227 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6"] Oct 30 18:34:49.353392 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.353354 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nvrz7\" (UniqueName: \"kubernetes.io/projected/59aec578-9be5-4f28-ac32-26270ee0f22e-kube-api-access-nvrz7\") pod \"maas-api-revoke-sub-85237e45-24pt6\" (UID: \"59aec578-9be5-4f28-ac32-26270ee0f22e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6" Oct 30 18:34:49.353392 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.353400 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59aec578-9be5-4f28-ac32-26270ee0f22e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-24pt6\" (UID: \"59aec578-9be5-4f28-ac32-26270ee0f22e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6" Oct 30 18:34:49.353916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.353495 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59aec578-9be5-4f28-ac32-26270ee0f22e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-24pt6\" (UID: \"59aec578-9be5-4f28-ac32-26270ee0f22e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6" Oct 30 18:34:49.460988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.457045 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59aec578-9be5-4f28-ac32-26270ee0f22e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-24pt6\" (UID: \"59aec578-9be5-4f28-ac32-26270ee0f22e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6" Oct 30 18:34:49.460988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.457185 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nvrz7\" (UniqueName: \"kubernetes.io/projected/59aec578-9be5-4f28-ac32-26270ee0f22e-kube-api-access-nvrz7\") pod \"maas-api-revoke-sub-85237e45-24pt6\" (UID: \"59aec578-9be5-4f28-ac32-26270ee0f22e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6" Oct 30 18:34:49.460988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.457233 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59aec578-9be5-4f28-ac32-26270ee0f22e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-24pt6\" (UID: \"59aec578-9be5-4f28-ac32-26270ee0f22e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6" Oct 30 18:34:49.460988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.458188 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59aec578-9be5-4f28-ac32-26270ee0f22e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-85237e45-24pt6\" (UID: \"59aec578-9be5-4f28-ac32-26270ee0f22e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6" Oct 30 18:34:49.463543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.461626 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59aec578-9be5-4f28-ac32-26270ee0f22e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-85237e45-24pt6\" (UID: \"59aec578-9be5-4f28-ac32-26270ee0f22e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6" Oct 30 18:34:49.485607 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.479449 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nvrz7\" (UniqueName: \"kubernetes.io/projected/59aec578-9be5-4f28-ac32-26270ee0f22e-kube-api-access-nvrz7\") pod \"maas-api-revoke-sub-85237e45-24pt6\" (UID: \"59aec578-9be5-4f28-ac32-26270ee0f22e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6" Oct 30 18:34:49.595210 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.595133 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6" Oct 30 18:34:49.669651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.669619 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:49.681539 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.681490 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:49.948582 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:49.947779 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6"] Oct 30 18:34:49.971865 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:34:49.971815 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod59aec578_9be5_4f28_ac32_26270ee0f22e.slice/crio-9317c3267f84b04ad67c929aa0ad7e281a5364b5420c5b5348a86fcfedce4cd1 WatchSource:0}: Error finding container 9317c3267f84b04ad67c929aa0ad7e281a5364b5420c5b5348a86fcfedce4cd1: Status 404 returned error can't find the container with id 9317c3267f84b04ad67c929aa0ad7e281a5364b5420c5b5348a86fcfedce4cd1 Oct 30 18:34:50.163683 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:50.160568 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6" event={"ID":"59aec578-9be5-4f28-ac32-26270ee0f22e","Type":"ContainerStarted","Data":"9317c3267f84b04ad67c929aa0ad7e281a5364b5420c5b5348a86fcfedce4cd1"} Oct 30 18:34:50.670831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:50.670717 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zzlvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 18:34:50.670831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:50.670786 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" podUID="b075648b-215b-4d0e-b3dc-41ea5ed46ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 18:34:50.683218 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:50.683184 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kbsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 18:34:50.683364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:50.683237 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" podUID="92403b9c-a622-457e-8b9b-a92f86f93b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 18:34:51.167807 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:51.167771 5232 generic.go:356] "Generic (PLEG): container finished" podID="59aec578-9be5-4f28-ac32-26270ee0f22e" containerID="7e0d40bf7f7b72d08bc908f1a74172d7f9f019eef60f39c1eda9c8d121a090a1" exitCode=0 Oct 30 18:34:51.168010 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:51.167859 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6" event={"ID":"59aec578-9be5-4f28-ac32-26270ee0f22e","Type":"ContainerDied","Data":"7e0d40bf7f7b72d08bc908f1a74172d7f9f019eef60f39c1eda9c8d121a090a1"} Oct 30 18:34:51.669073 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:51.669039 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zzlvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 18:34:51.669314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:51.669293 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" podUID="b075648b-215b-4d0e-b3dc-41ea5ed46ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 18:34:51.681963 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:51.681930 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kbsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 18:34:51.682345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:51.681993 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" podUID="92403b9c-a622-457e-8b9b-a92f86f93b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 18:34:52.385326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:52.385297 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6" Oct 30 18:34:52.487263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:52.487224 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59aec578-9be5-4f28-ac32-26270ee0f22e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59aec578-9be5-4f28-ac32-26270ee0f22e-maas-api-openshift-ca\") pod \"59aec578-9be5-4f28-ac32-26270ee0f22e\" (UID: \"59aec578-9be5-4f28-ac32-26270ee0f22e\") " Oct 30 18:34:52.487465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:52.487285 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59aec578-9be5-4f28-ac32-26270ee0f22e-kube-api-access-nvrz7\" (UniqueName: \"kubernetes.io/projected/59aec578-9be5-4f28-ac32-26270ee0f22e-kube-api-access-nvrz7\") pod \"59aec578-9be5-4f28-ac32-26270ee0f22e\" (UID: \"59aec578-9be5-4f28-ac32-26270ee0f22e\") " Oct 30 18:34:52.487465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:52.487350 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/59aec578-9be5-4f28-ac32-26270ee0f22e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59aec578-9be5-4f28-ac32-26270ee0f22e-maas-api-xks-ca\") pod \"59aec578-9be5-4f28-ac32-26270ee0f22e\" (UID: \"59aec578-9be5-4f28-ac32-26270ee0f22e\") " Oct 30 18:34:52.487773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:52.487724 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59aec578-9be5-4f28-ac32-26270ee0f22e-maas-api-openshift-ca" pod "59aec578-9be5-4f28-ac32-26270ee0f22e" (UID: "59aec578-9be5-4f28-ac32-26270ee0f22e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:52.489636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:52.489611 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59aec578-9be5-4f28-ac32-26270ee0f22e-kube-api-access-nvrz7" pod "59aec578-9be5-4f28-ac32-26270ee0f22e" (UID: "59aec578-9be5-4f28-ac32-26270ee0f22e"). InnerVolumeSpecName "kube-api-access-nvrz7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:52.489744 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:52.489712 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/59aec578-9be5-4f28-ac32-26270ee0f22e-maas-api-xks-ca" pod "59aec578-9be5-4f28-ac32-26270ee0f22e" (UID: "59aec578-9be5-4f28-ac32-26270ee0f22e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:52.589085 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:52.589048 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/59aec578-9be5-4f28-ac32-26270ee0f22e-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:52.589085 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:52.589084 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/59aec578-9be5-4f28-ac32-26270ee0f22e-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:52.589297 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:52.589102 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nvrz7\" (UniqueName: \"kubernetes.io/projected/59aec578-9be5-4f28-ac32-26270ee0f22e-kube-api-access-nvrz7\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:52.669230 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:52.669185 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zzlvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 18:34:52.669437 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:52.669246 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" podUID="b075648b-215b-4d0e-b3dc-41ea5ed46ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 18:34:52.682594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:52.682557 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kbsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 18:34:52.682958 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:52.682613 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" podUID="92403b9c-a622-457e-8b9b-a92f86f93b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 18:34:53.181036 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:53.180942 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6" Oct 30 18:34:53.181036 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:53.180966 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6" event={"ID":"59aec578-9be5-4f28-ac32-26270ee0f22e","Type":"ContainerDied","Data":"9317c3267f84b04ad67c929aa0ad7e281a5364b5420c5b5348a86fcfedce4cd1"} Oct 30 18:34:53.181036 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:53.181014 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9317c3267f84b04ad67c929aa0ad7e281a5364b5420c5b5348a86fcfedce4cd1" Oct 30 18:34:53.669217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:53.669181 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zzlvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 18:34:53.669871 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:53.669480 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" podUID="b075648b-215b-4d0e-b3dc-41ea5ed46ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 18:34:53.681657 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:53.681631 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kbsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 18:34:53.681765 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:53.681673 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" podUID="92403b9c-a622-457e-8b9b-a92f86f93b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 18:34:54.669090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:54.669053 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zzlvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 18:34:54.669537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:54.669109 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" podUID="b075648b-215b-4d0e-b3dc-41ea5ed46ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 18:34:54.682134 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:54.682105 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kbsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 18:34:54.682237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:54.682157 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" podUID="92403b9c-a622-457e-8b9b-a92f86f93b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 18:34:55.146882 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:55.146848 5232 scope.go:122] "RemoveContainer" containerID="22dd2addab4a7884dfb549cc122bba839e7ff56ca9b588b4196134ec562fa8d2" Oct 30 18:34:55.157229 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:55.157201 5232 scope.go:122] "RemoveContainer" containerID="6d40f668e435acf97f6a97b17e6f88669431cd68c6ee6eb5c0ac74c6de85029f" Oct 30 18:34:55.167783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:55.167759 5232 scope.go:122] "RemoveContainer" containerID="923e86a84ff85061a5bce8061ef77980795d144addff6a42a903a609e0c8bb38" Oct 30 18:34:55.177205 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:55.177185 5232 scope.go:122] "RemoveContainer" containerID="1074582ce2c8e166e3c40d9f09200b727bce7809c4fd18579e5843257480fabf" Oct 30 18:34:55.185348 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:55.185324 5232 scope.go:122] "RemoveContainer" containerID="81357893498d7486f1c0e4f5915aea050ee6a85e19dfd5776e156d40cb4e0c31" Oct 30 18:34:55.194854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:55.194831 5232 scope.go:122] "RemoveContainer" containerID="ae1d8a6fb4f0b3a0d5d1fb709fd76eda8c155b90cd123db9659dd15de8015eb8" Oct 30 18:34:55.203564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:55.203484 5232 scope.go:122] "RemoveContainer" containerID="edbed106988a4d3f27fb19337915a3a275f66207b996e5c53128666ed40211b7" Oct 30 18:34:55.668364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:55.668327 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zzlvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" start-of-body= Oct 30 18:34:55.668598 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:55.668388 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" podUID="b075648b-215b-4d0e-b3dc-41ea5ed46ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": dial tcp 10.133.0.246:15021: connect: connection refused" Oct 30 18:34:55.681678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:55.681645 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kbsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 18:34:55.682179 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:55.681705 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" podUID="92403b9c-a622-457e-8b9b-a92f86f93b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 18:34:56.262824 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.262789 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp"] Oct 30 18:34:56.263004 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.262966 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" podUID="322b3612-668d-4d50-8506-9df154d5e57d" containerName="discovery" containerID="cri-o://4765120e0dbf004d1b47886aac1e935164e16ae8a81bd66c9e50b1dbe2f86f41" gracePeriod=30 Oct 30 18:34:56.545779 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.545736 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:34:56.637614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.637580 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/322b3612-668d-4d50-8506-9df154d5e57d-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/322b3612-668d-4d50-8506-9df154d5e57d-istio-csr-ca-configmap\") pod \"322b3612-668d-4d50-8506-9df154d5e57d\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " Oct 30 18:34:56.637614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.637616 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-istio-csr-dns-cert\") pod \"322b3612-668d-4d50-8506-9df154d5e57d\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " Oct 30 18:34:56.637844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.637643 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322b3612-668d-4d50-8506-9df154d5e57d-kube-api-access-qt4fm\" (UniqueName: \"kubernetes.io/projected/322b3612-668d-4d50-8506-9df154d5e57d-kube-api-access-qt4fm\") pod \"322b3612-668d-4d50-8506-9df154d5e57d\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " Oct 30 18:34:56.637844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.637687 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-cacerts\" (UniqueName: \"kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-cacerts\") pod \"322b3612-668d-4d50-8506-9df154d5e57d\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " Oct 30 18:34:56.637844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.637747 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-istio-kubeconfig\") pod \"322b3612-668d-4d50-8506-9df154d5e57d\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " Oct 30 18:34:56.637844 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.637786 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/322b3612-668d-4d50-8506-9df154d5e57d-istio-token\" (UniqueName: \"kubernetes.io/projected/322b3612-668d-4d50-8506-9df154d5e57d-istio-token\") pod \"322b3612-668d-4d50-8506-9df154d5e57d\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " Oct 30 18:34:56.638066 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.637867 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/322b3612-668d-4d50-8506-9df154d5e57d-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/322b3612-668d-4d50-8506-9df154d5e57d-local-certs\") pod \"322b3612-668d-4d50-8506-9df154d5e57d\" (UID: \"322b3612-668d-4d50-8506-9df154d5e57d\") " Oct 30 18:34:56.638128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.638104 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/322b3612-668d-4d50-8506-9df154d5e57d-istio-csr-ca-configmap" pod "322b3612-668d-4d50-8506-9df154d5e57d" (UID: "322b3612-668d-4d50-8506-9df154d5e57d"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:34:56.638294 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.638269 5232 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/322b3612-668d-4d50-8506-9df154d5e57d-istio-csr-ca-configmap\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:56.640999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.640877 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322b3612-668d-4d50-8506-9df154d5e57d-kube-api-access-qt4fm" pod "322b3612-668d-4d50-8506-9df154d5e57d" (UID: "322b3612-668d-4d50-8506-9df154d5e57d"). InnerVolumeSpecName "kube-api-access-qt4fm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:56.640999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.640971 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-istio-csr-dns-cert" pod "322b3612-668d-4d50-8506-9df154d5e57d" (UID: "322b3612-668d-4d50-8506-9df154d5e57d"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:56.641199 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.641081 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/322b3612-668d-4d50-8506-9df154d5e57d-istio-token" pod "322b3612-668d-4d50-8506-9df154d5e57d" (UID: "322b3612-668d-4d50-8506-9df154d5e57d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:34:56.641199 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.641181 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-istio-kubeconfig" pod "322b3612-668d-4d50-8506-9df154d5e57d" (UID: "322b3612-668d-4d50-8506-9df154d5e57d"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:56.641538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.641492 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/322b3612-668d-4d50-8506-9df154d5e57d-local-certs" pod "322b3612-668d-4d50-8506-9df154d5e57d" (UID: "322b3612-668d-4d50-8506-9df154d5e57d"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:34:56.641640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.641579 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-cacerts" pod "322b3612-668d-4d50-8506-9df154d5e57d" (UID: "322b3612-668d-4d50-8506-9df154d5e57d"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:34:56.681880 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.681835 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kbsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" start-of-body= Oct 30 18:34:56.682405 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.681919 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" podUID="92403b9c-a622-457e-8b9b-a92f86f93b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": dial tcp 10.133.0.247:15021: connect: connection refused" Oct 30 18:34:56.739683 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.739637 5232 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/322b3612-668d-4d50-8506-9df154d5e57d-local-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:56.739683 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.739679 5232 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-istio-csr-dns-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:56.739964 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.739695 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qt4fm\" (UniqueName: \"kubernetes.io/projected/322b3612-668d-4d50-8506-9df154d5e57d-kube-api-access-qt4fm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:56.739964 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.739711 5232 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-cacerts\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:56.739964 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.739726 5232 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/322b3612-668d-4d50-8506-9df154d5e57d-istio-kubeconfig\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:56.739964 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:56.739740 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/322b3612-668d-4d50-8506-9df154d5e57d-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:34:57.047197 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:34:57.047153 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:34:57.214896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:57.214794 5232 generic.go:356] "Generic (PLEG): container finished" podID="322b3612-668d-4d50-8506-9df154d5e57d" containerID="4765120e0dbf004d1b47886aac1e935164e16ae8a81bd66c9e50b1dbe2f86f41" exitCode=0 Oct 30 18:34:57.214896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:57.214854 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" Oct 30 18:34:57.215134 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:57.214978 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" event={"ID":"322b3612-668d-4d50-8506-9df154d5e57d","Type":"ContainerDied","Data":"4765120e0dbf004d1b47886aac1e935164e16ae8a81bd66c9e50b1dbe2f86f41"} Oct 30 18:34:57.215134 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:57.215025 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp" event={"ID":"322b3612-668d-4d50-8506-9df154d5e57d","Type":"ContainerDied","Data":"278e90aae64dbcfec5ef1203757e5ca65950092357c28f7a41e51c717eba54b2"} Oct 30 18:34:57.215134 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:57.215054 5232 scope.go:122] "RemoveContainer" containerID="4765120e0dbf004d1b47886aac1e935164e16ae8a81bd66c9e50b1dbe2f86f41" Oct 30 18:34:57.239181 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:57.239150 5232 scope.go:122] "RemoveContainer" containerID="4765120e0dbf004d1b47886aac1e935164e16ae8a81bd66c9e50b1dbe2f86f41" Oct 30 18:34:57.239559 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:34:57.239501 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4765120e0dbf004d1b47886aac1e935164e16ae8a81bd66c9e50b1dbe2f86f41\": container with ID starting with 4765120e0dbf004d1b47886aac1e935164e16ae8a81bd66c9e50b1dbe2f86f41 not found: ID does not exist" containerID="4765120e0dbf004d1b47886aac1e935164e16ae8a81bd66c9e50b1dbe2f86f41" Oct 30 18:34:57.239744 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:57.239574 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4765120e0dbf004d1b47886aac1e935164e16ae8a81bd66c9e50b1dbe2f86f41"} err="failed to get container status \"4765120e0dbf004d1b47886aac1e935164e16ae8a81bd66c9e50b1dbe2f86f41\": rpc error: code = NotFound desc = could not find container \"4765120e0dbf004d1b47886aac1e935164e16ae8a81bd66c9e50b1dbe2f86f41\": container with ID starting with 4765120e0dbf004d1b47886aac1e935164e16ae8a81bd66c9e50b1dbe2f86f41 not found: ID does not exist" Oct 30 18:34:57.279708 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:57.279673 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp"] Oct 30 18:34:57.291867 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:57.291832 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-t7ntp"] Oct 30 18:34:57.670046 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:57.670015 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zzlvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:34:57.670263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:57.670070 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" podUID="b075648b-215b-4d0e-b3dc-41ea5ed46ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:34:58.711742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:58.711692 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zzlvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:34:58.712380 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:58.711773 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" podUID="b075648b-215b-4d0e-b3dc-41ea5ed46ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:34:58.722705 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:58.722659 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kbsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:34:58.722959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:58.722918 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" podUID="92403b9c-a622-457e-8b9b-a92f86f93b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:34:58.800894 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:58.800862 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="322b3612-668d-4d50-8506-9df154d5e57d" path="/var/lib/kubelet/pods/322b3612-668d-4d50-8506-9df154d5e57d/volumes" Oct 30 18:34:59.753723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:59.753683 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zzlvr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:34:59.754240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:59.753749 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" podUID="b075648b-215b-4d0e-b3dc-41ea5ed46ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:34:59.764691 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:59.764650 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kbsdt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:34:59.764847 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:59.764707 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" podUID="92403b9c-a622-457e-8b9b-a92f86f93b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:34:59.778540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:59.778494 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:59.778803 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:59.778786 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:59.779596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:59.779575 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" Oct 30 18:34:59.988765 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:59.988734 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:59.989055 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:59.989014 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:34:59.989730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:34:59.989711 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" Oct 30 18:35:01.092955 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:01.092921 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c97f9884-qdl7l"] Oct 30 18:35:01.093378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:01.093149 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-68c97f9884-qdl7l" podUID="bb4dfa63-93b7-4a41-bf1b-1b9151b60655" containerName="manager" containerID="cri-o://a10dacdced632b5772fb63dea1e1fe1c1346b86fd772d1caec4c1ecf7e55a7e7" gracePeriod=30 Oct 30 18:35:01.249568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:01.249499 5232 generic.go:356] "Generic (PLEG): container finished" podID="bb4dfa63-93b7-4a41-bf1b-1b9151b60655" containerID="a10dacdced632b5772fb63dea1e1fe1c1346b86fd772d1caec4c1ecf7e55a7e7" exitCode=0 Oct 30 18:35:01.249568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:01.249537 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c97f9884-qdl7l" event={"ID":"bb4dfa63-93b7-4a41-bf1b-1b9151b60655","Type":"ContainerDied","Data":"a10dacdced632b5772fb63dea1e1fe1c1346b86fd772d1caec4c1ecf7e55a7e7"} Oct 30 18:35:01.351113 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:01.351058 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c97f9884-qdl7l" Oct 30 18:35:01.391453 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:01.391423 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-kube-api-access-d4hmp\" (UniqueName: \"kubernetes.io/projected/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-kube-api-access-d4hmp\") pod \"bb4dfa63-93b7-4a41-bf1b-1b9151b60655\" (UID: \"bb4dfa63-93b7-4a41-bf1b-1b9151b60655\") " Oct 30 18:35:01.391665 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:01.391587 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-metrics-certs\") pod \"bb4dfa63-93b7-4a41-bf1b-1b9151b60655\" (UID: \"bb4dfa63-93b7-4a41-bf1b-1b9151b60655\") " Oct 30 18:35:01.391665 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:01.391650 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-webhook-cert\") pod \"bb4dfa63-93b7-4a41-bf1b-1b9151b60655\" (UID: \"bb4dfa63-93b7-4a41-bf1b-1b9151b60655\") " Oct 30 18:35:01.394754 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:01.394721 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-webhook-cert" pod "bb4dfa63-93b7-4a41-bf1b-1b9151b60655" (UID: "bb4dfa63-93b7-4a41-bf1b-1b9151b60655"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:35:01.394996 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:01.394975 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-kube-api-access-d4hmp" pod "bb4dfa63-93b7-4a41-bf1b-1b9151b60655" (UID: "bb4dfa63-93b7-4a41-bf1b-1b9151b60655"). InnerVolumeSpecName "kube-api-access-d4hmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:01.395108 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:01.395088 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-metrics-certs" pod "bb4dfa63-93b7-4a41-bf1b-1b9151b60655" (UID: "bb4dfa63-93b7-4a41-bf1b-1b9151b60655"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:35:01.493622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:01.493585 5232 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-metrics-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:35:01.493622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:01.493616 5232 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-webhook-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:35:01.493622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:01.493625 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4hmp\" (UniqueName: \"kubernetes.io/projected/bb4dfa63-93b7-4a41-bf1b-1b9151b60655-kube-api-access-d4hmp\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:35:02.255592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:02.255554 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c97f9884-qdl7l" event={"ID":"bb4dfa63-93b7-4a41-bf1b-1b9151b60655","Type":"ContainerDied","Data":"4cbaf3779b6f2d32f87597b4b85560434034bf43ca55a974698fcae88ab001f3"} Oct 30 18:35:02.255592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:02.255575 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c97f9884-qdl7l" Oct 30 18:35:02.256139 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:02.255611 5232 scope.go:122] "RemoveContainer" containerID="a10dacdced632b5772fb63dea1e1fe1c1346b86fd772d1caec4c1ecf7e55a7e7" Oct 30 18:35:02.304809 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:02.304776 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-68c97f9884-qdl7l"] Oct 30 18:35:02.307548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:02.307527 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-68c97f9884-qdl7l"] Oct 30 18:35:02.801168 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:02.801131 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb4dfa63-93b7-4a41-bf1b-1b9151b60655" path="/var/lib/kubelet/pods/bb4dfa63-93b7-4a41-bf1b-1b9151b60655/volumes" Oct 30 18:35:03.617973 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.617944 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-68c97f9884-jbsf4"] Oct 30 18:35:03.618382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.618282 5232 cpu_manager.go:414] "containerMap: removing container" podUID="59aec578-9be5-4f28-ac32-26270ee0f22e" containerName="revoke-keys" Oct 30 18:35:03.618382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.618292 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59aec578-9be5-4f28-ac32-26270ee0f22e" containerName="revoke-keys" Oct 30 18:35:03.618382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.618316 5232 cpu_manager.go:414] "containerMap: removing container" podUID="322b3612-668d-4d50-8506-9df154d5e57d" containerName="discovery" Oct 30 18:35:03.618382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.618321 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="322b3612-668d-4d50-8506-9df154d5e57d" containerName="discovery" Oct 30 18:35:03.618382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.618334 5232 cpu_manager.go:414] "containerMap: removing container" podUID="bb4dfa63-93b7-4a41-bf1b-1b9151b60655" containerName="manager" Oct 30 18:35:03.618382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.618339 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb4dfa63-93b7-4a41-bf1b-1b9151b60655" containerName="manager" Oct 30 18:35:03.618615 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.618387 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="322b3612-668d-4d50-8506-9df154d5e57d" containerName="discovery" Oct 30 18:35:03.618615 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.618400 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb4dfa63-93b7-4a41-bf1b-1b9151b60655" containerName="manager" Oct 30 18:35:03.618615 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.618408 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="59aec578-9be5-4f28-ac32-26270ee0f22e" containerName="revoke-keys" Oct 30 18:35:03.621629 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.621610 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c97f9884-jbsf4" Oct 30 18:35:03.640465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.640443 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 18:35:03.650365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.650344 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-r2cqt\"" Oct 30 18:35:03.650487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.650392 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 30 18:35:03.650487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.650401 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 30 18:35:03.650487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.650458 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 30 18:35:03.650487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.650470 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 18:35:03.654320 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.654298 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c97f9884-jbsf4"] Oct 30 18:35:03.712651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.712628 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2000f04f-227c-415d-bc3a-d270dda9e0c6-metrics-certs\") pod \"maas-controller-68c97f9884-jbsf4\" (UID: \"2000f04f-227c-415d-bc3a-d270dda9e0c6\") " pod="opendatahub/maas-controller-68c97f9884-jbsf4" Oct 30 18:35:03.712792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.712673 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8stj7\" (UniqueName: \"kubernetes.io/projected/2000f04f-227c-415d-bc3a-d270dda9e0c6-kube-api-access-8stj7\") pod \"maas-controller-68c97f9884-jbsf4\" (UID: \"2000f04f-227c-415d-bc3a-d270dda9e0c6\") " pod="opendatahub/maas-controller-68c97f9884-jbsf4" Oct 30 18:35:03.712792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.712753 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2000f04f-227c-415d-bc3a-d270dda9e0c6-webhook-cert\") pod \"maas-controller-68c97f9884-jbsf4\" (UID: \"2000f04f-227c-415d-bc3a-d270dda9e0c6\") " pod="opendatahub/maas-controller-68c97f9884-jbsf4" Oct 30 18:35:03.814058 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.814017 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8stj7\" (UniqueName: \"kubernetes.io/projected/2000f04f-227c-415d-bc3a-d270dda9e0c6-kube-api-access-8stj7\") pod \"maas-controller-68c97f9884-jbsf4\" (UID: \"2000f04f-227c-415d-bc3a-d270dda9e0c6\") " pod="opendatahub/maas-controller-68c97f9884-jbsf4" Oct 30 18:35:03.814208 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.814103 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2000f04f-227c-415d-bc3a-d270dda9e0c6-webhook-cert\") pod \"maas-controller-68c97f9884-jbsf4\" (UID: \"2000f04f-227c-415d-bc3a-d270dda9e0c6\") " pod="opendatahub/maas-controller-68c97f9884-jbsf4" Oct 30 18:35:03.814208 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.814170 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2000f04f-227c-415d-bc3a-d270dda9e0c6-metrics-certs\") pod \"maas-controller-68c97f9884-jbsf4\" (UID: \"2000f04f-227c-415d-bc3a-d270dda9e0c6\") " pod="opendatahub/maas-controller-68c97f9884-jbsf4" Oct 30 18:35:03.816649 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.816624 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2000f04f-227c-415d-bc3a-d270dda9e0c6-metrics-certs\") pod \"maas-controller-68c97f9884-jbsf4\" (UID: \"2000f04f-227c-415d-bc3a-d270dda9e0c6\") " pod="opendatahub/maas-controller-68c97f9884-jbsf4" Oct 30 18:35:03.816760 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.816727 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2000f04f-227c-415d-bc3a-d270dda9e0c6-webhook-cert\") pod \"maas-controller-68c97f9884-jbsf4\" (UID: \"2000f04f-227c-415d-bc3a-d270dda9e0c6\") " pod="opendatahub/maas-controller-68c97f9884-jbsf4" Oct 30 18:35:03.840225 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.840200 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8stj7\" (UniqueName: \"kubernetes.io/projected/2000f04f-227c-415d-bc3a-d270dda9e0c6-kube-api-access-8stj7\") pod \"maas-controller-68c97f9884-jbsf4\" (UID: \"2000f04f-227c-415d-bc3a-d270dda9e0c6\") " pod="opendatahub/maas-controller-68c97f9884-jbsf4" Oct 30 18:35:03.933466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:03.933396 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-68c97f9884-jbsf4" Oct 30 18:35:04.023226 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:04.023196 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22"] Oct 30 18:35:04.028137 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:04.028112 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6fbc0906-dvr22"] Oct 30 18:35:04.234132 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:04.234103 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-68c97f9884-jbsf4"] Oct 30 18:35:04.236109 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:35:04.236081 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2000f04f_227c_415d_bc3a_d270dda9e0c6.slice/crio-9e041d31251ab3e6ed955e93287463f77c9851f169ee9dd1c1d6ff4ae6ea96c5 WatchSource:0}: Error finding container 9e041d31251ab3e6ed955e93287463f77c9851f169ee9dd1c1d6ff4ae6ea96c5: Status 404 returned error can't find the container with id 9e041d31251ab3e6ed955e93287463f77c9851f169ee9dd1c1d6ff4ae6ea96c5 Oct 30 18:35:04.269073 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:04.269040 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c97f9884-jbsf4" event={"ID":"2000f04f-227c-415d-bc3a-d270dda9e0c6","Type":"ContainerStarted","Data":"9e041d31251ab3e6ed955e93287463f77c9851f169ee9dd1c1d6ff4ae6ea96c5"} Oct 30 18:35:04.801120 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:04.801075 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="391ee642-1ab6-47cc-b103-f381ea9fd5bd" path="/var/lib/kubelet/pods/391ee642-1ab6-47cc-b103-f381ea9fd5bd/volumes" Oct 30 18:35:05.276049 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:05.276010 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-68c97f9884-jbsf4" event={"ID":"2000f04f-227c-415d-bc3a-d270dda9e0c6","Type":"ContainerStarted","Data":"655324bec13312368ed9d4fc1301244b74559b5529760326319cf31c86bb442c"} Oct 30 18:35:05.276224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:05.276074 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-68c97f9884-jbsf4" Oct 30 18:35:05.317141 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:05.317092 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-68c97f9884-jbsf4" podStartSLOduration=1.957549453 podStartE2EDuration="2.317077416s" totalImagesPullingTime="359.527963ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:35:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:35:05.315810048 +0000 UTC m=+2004.632230368" watchObservedRunningTime="2026-10-30 18:35:05.317077416 +0000 UTC m=+2004.633497737" Oct 30 18:35:16.283771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:16.283737 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-68c97f9884-jbsf4" Oct 30 18:35:27.032787 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:35:27.032744 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:35:54.860978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:54.860943 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5"] Oct 30 18:35:54.864695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:54.864674 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5" Oct 30 18:35:54.893718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:54.893682 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:35:54.893995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:54.893963 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:35:54.895875 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:54.895855 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5"] Oct 30 18:35:55.015094 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.015065 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b32b717-279b-4232-a275-d6ac72d86937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dvfw5\" (UID: \"8b32b717-279b-4232-a275-d6ac72d86937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5" Oct 30 18:35:55.015289 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.015104 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k8bx9\" (UniqueName: \"kubernetes.io/projected/8b32b717-279b-4232-a275-d6ac72d86937-kube-api-access-k8bx9\") pod \"maas-api-revoke-sub-c1e271a9-dvfw5\" (UID: \"8b32b717-279b-4232-a275-d6ac72d86937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5" Oct 30 18:35:55.015289 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.015130 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b32b717-279b-4232-a275-d6ac72d86937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dvfw5\" (UID: \"8b32b717-279b-4232-a275-d6ac72d86937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5" Oct 30 18:35:55.116176 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.116095 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b32b717-279b-4232-a275-d6ac72d86937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dvfw5\" (UID: \"8b32b717-279b-4232-a275-d6ac72d86937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5" Oct 30 18:35:55.116176 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.116134 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k8bx9\" (UniqueName: \"kubernetes.io/projected/8b32b717-279b-4232-a275-d6ac72d86937-kube-api-access-k8bx9\") pod \"maas-api-revoke-sub-c1e271a9-dvfw5\" (UID: \"8b32b717-279b-4232-a275-d6ac72d86937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5" Oct 30 18:35:55.116176 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.116160 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b32b717-279b-4232-a275-d6ac72d86937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dvfw5\" (UID: \"8b32b717-279b-4232-a275-d6ac72d86937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5" Oct 30 18:35:55.116775 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.116754 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b32b717-279b-4232-a275-d6ac72d86937-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c1e271a9-dvfw5\" (UID: \"8b32b717-279b-4232-a275-d6ac72d86937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5" Oct 30 18:35:55.118693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.118671 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b32b717-279b-4232-a275-d6ac72d86937-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c1e271a9-dvfw5\" (UID: \"8b32b717-279b-4232-a275-d6ac72d86937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5" Oct 30 18:35:55.141855 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.141831 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k8bx9\" (UniqueName: \"kubernetes.io/projected/8b32b717-279b-4232-a275-d6ac72d86937-kube-api-access-k8bx9\") pod \"maas-api-revoke-sub-c1e271a9-dvfw5\" (UID: \"8b32b717-279b-4232-a275-d6ac72d86937\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5" Oct 30 18:35:55.177096 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.177076 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5" Oct 30 18:35:55.278817 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.278788 5232 scope.go:122] "RemoveContainer" containerID="10e77edb2df73cdb870bcd223e9b0f683be5da908a21c1e7564b6185c076322c" Oct 30 18:35:55.287752 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.287731 5232 scope.go:122] "RemoveContainer" containerID="ed1575ed85c1db9547135d18971d4920b6fff46ef29ccfbf04c823180eae59da" Oct 30 18:35:55.295776 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.295758 5232 scope.go:122] "RemoveContainer" containerID="f635c709337dde3b11a5a79276e62fa9b4b368edbd455d2266d8b16c94bd143a" Oct 30 18:35:55.303622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.303604 5232 scope.go:122] "RemoveContainer" containerID="b751c94e182307ab1a30516d1000d50547d0fb11d9719d4c3f1f571150d86f5a" Oct 30 18:35:55.311401 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.311373 5232 scope.go:122] "RemoveContainer" containerID="be247825d58d37067a528577f5cf3bf1be1d100c4221c3b2b17d8506607795d4" Oct 30 18:35:55.319325 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.319305 5232 scope.go:122] "RemoveContainer" containerID="463045d630af32830791c9940546a65573f3204f5e1d29ec4934fbf40f3863ec" Oct 30 18:35:55.327062 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.327046 5232 scope.go:122] "RemoveContainer" containerID="a4108ce01f8ce085eb19b6999f585c836690d14f74e7f044cb0e29cfa5bc1106" Oct 30 18:35:55.476249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.474971 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5"] Oct 30 18:35:55.479564 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:35:55.479502 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b32b717_279b_4232_a275_d6ac72d86937.slice/crio-6b584bc0b44b99591182ebecf813fdf10b8be5729b04bc26bfcb5f8c7c302c1c WatchSource:0}: Error finding container 6b584bc0b44b99591182ebecf813fdf10b8be5729b04bc26bfcb5f8c7c302c1c: Status 404 returned error can't find the container with id 6b584bc0b44b99591182ebecf813fdf10b8be5729b04bc26bfcb5f8c7c302c1c Oct 30 18:35:55.565567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:55.565532 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5" event={"ID":"8b32b717-279b-4232-a275-d6ac72d86937","Type":"ContainerStarted","Data":"6b584bc0b44b99591182ebecf813fdf10b8be5729b04bc26bfcb5f8c7c302c1c"} Oct 30 18:35:56.571936 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:56.571901 5232 generic.go:356] "Generic (PLEG): container finished" podID="8b32b717-279b-4232-a275-d6ac72d86937" containerID="de457d1690f24c25ca03aaea246614753f3bd07773c0bd8e8a25cb14e0537fae" exitCode=0 Oct 30 18:35:56.572359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:56.571957 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5" event={"ID":"8b32b717-279b-4232-a275-d6ac72d86937","Type":"ContainerDied","Data":"de457d1690f24c25ca03aaea246614753f3bd07773c0bd8e8a25cb14e0537fae"} Oct 30 18:35:57.032655 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:35:57.032621 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:35:57.700556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:57.700532 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5" Oct 30 18:35:57.837730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:57.837662 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b32b717-279b-4232-a275-d6ac72d86937-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b32b717-279b-4232-a275-d6ac72d86937-maas-api-openshift-ca\") pod \"8b32b717-279b-4232-a275-d6ac72d86937\" (UID: \"8b32b717-279b-4232-a275-d6ac72d86937\") " Oct 30 18:35:57.837730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:57.837702 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b32b717-279b-4232-a275-d6ac72d86937-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b32b717-279b-4232-a275-d6ac72d86937-maas-api-xks-ca\") pod \"8b32b717-279b-4232-a275-d6ac72d86937\" (UID: \"8b32b717-279b-4232-a275-d6ac72d86937\") " Oct 30 18:35:57.837916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:57.837759 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b32b717-279b-4232-a275-d6ac72d86937-kube-api-access-k8bx9\" (UniqueName: \"kubernetes.io/projected/8b32b717-279b-4232-a275-d6ac72d86937-kube-api-access-k8bx9\") pod \"8b32b717-279b-4232-a275-d6ac72d86937\" (UID: \"8b32b717-279b-4232-a275-d6ac72d86937\") " Oct 30 18:35:57.838025 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:57.838004 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b32b717-279b-4232-a275-d6ac72d86937-maas-api-openshift-ca" pod "8b32b717-279b-4232-a275-d6ac72d86937" (UID: "8b32b717-279b-4232-a275-d6ac72d86937"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:35:57.840006 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:57.839982 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b32b717-279b-4232-a275-d6ac72d86937-maas-api-xks-ca" pod "8b32b717-279b-4232-a275-d6ac72d86937" (UID: "8b32b717-279b-4232-a275-d6ac72d86937"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:35:57.840057 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:57.840018 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b32b717-279b-4232-a275-d6ac72d86937-kube-api-access-k8bx9" pod "8b32b717-279b-4232-a275-d6ac72d86937" (UID: "8b32b717-279b-4232-a275-d6ac72d86937"). InnerVolumeSpecName "kube-api-access-k8bx9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:35:57.939174 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:57.939089 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k8bx9\" (UniqueName: \"kubernetes.io/projected/8b32b717-279b-4232-a275-d6ac72d86937-kube-api-access-k8bx9\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:35:57.939174 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:57.939110 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b32b717-279b-4232-a275-d6ac72d86937-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:35:57.939174 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:57.939119 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b32b717-279b-4232-a275-d6ac72d86937-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:35:58.582912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:58.582869 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5" Oct 30 18:35:58.582912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:58.582897 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5" event={"ID":"8b32b717-279b-4232-a275-d6ac72d86937","Type":"ContainerDied","Data":"6b584bc0b44b99591182ebecf813fdf10b8be5729b04bc26bfcb5f8c7c302c1c"} Oct 30 18:35:58.583129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:35:58.582955 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6b584bc0b44b99591182ebecf813fdf10b8be5729b04bc26bfcb5f8c7c302c1c" Oct 30 18:36:21.198989 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.198914 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp"] Oct 30 18:36:21.199530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.199349 5232 cpu_manager.go:414] "containerMap: removing container" podUID="8b32b717-279b-4232-a275-d6ac72d86937" containerName="revoke-keys" Oct 30 18:36:21.199530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.199364 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b32b717-279b-4232-a275-d6ac72d86937" containerName="revoke-keys" Oct 30 18:36:21.199530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.199426 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b32b717-279b-4232-a275-d6ac72d86937" containerName="revoke-keys" Oct 30 18:36:21.202570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.202542 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp" Oct 30 18:36:21.231984 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.231953 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:36:21.232177 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.232158 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:36:21.234304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.234270 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp"] Oct 30 18:36:21.286344 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.286313 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t9287\" (UniqueName: \"kubernetes.io/projected/8aee1942-38a1-42af-9826-c316b3363258-kube-api-access-t9287\") pod \"maas-api-revoke-sub-2e1f5b5f-g8pzp\" (UID: \"8aee1942-38a1-42af-9826-c316b3363258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp" Oct 30 18:36:21.286565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.286362 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aee1942-38a1-42af-9826-c316b3363258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g8pzp\" (UID: \"8aee1942-38a1-42af-9826-c316b3363258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp" Oct 30 18:36:21.286565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.286382 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aee1942-38a1-42af-9826-c316b3363258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g8pzp\" (UID: \"8aee1942-38a1-42af-9826-c316b3363258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp" Oct 30 18:36:21.387200 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.387168 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t9287\" (UniqueName: \"kubernetes.io/projected/8aee1942-38a1-42af-9826-c316b3363258-kube-api-access-t9287\") pod \"maas-api-revoke-sub-2e1f5b5f-g8pzp\" (UID: \"8aee1942-38a1-42af-9826-c316b3363258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp" Oct 30 18:36:21.387396 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.387214 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aee1942-38a1-42af-9826-c316b3363258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g8pzp\" (UID: \"8aee1942-38a1-42af-9826-c316b3363258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp" Oct 30 18:36:21.387396 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.387235 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aee1942-38a1-42af-9826-c316b3363258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g8pzp\" (UID: \"8aee1942-38a1-42af-9826-c316b3363258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp" Oct 30 18:36:21.387876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.387849 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aee1942-38a1-42af-9826-c316b3363258-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g8pzp\" (UID: \"8aee1942-38a1-42af-9826-c316b3363258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp" Oct 30 18:36:21.389833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.389805 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aee1942-38a1-42af-9826-c316b3363258-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-g8pzp\" (UID: \"8aee1942-38a1-42af-9826-c316b3363258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp" Oct 30 18:36:21.407010 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.406981 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t9287\" (UniqueName: \"kubernetes.io/projected/8aee1942-38a1-42af-9826-c316b3363258-kube-api-access-t9287\") pod \"maas-api-revoke-sub-2e1f5b5f-g8pzp\" (UID: \"8aee1942-38a1-42af-9826-c316b3363258\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp" Oct 30 18:36:21.515109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.515077 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp" Oct 30 18:36:21.804789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:21.804767 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp"] Oct 30 18:36:21.806739 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:36:21.806712 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8aee1942_38a1_42af_9826_c316b3363258.slice/crio-79da11bb1735ddbffec90dea3537d85f3979213c07f1d8eb03d942694ff06414 WatchSource:0}: Error finding container 79da11bb1735ddbffec90dea3537d85f3979213c07f1d8eb03d942694ff06414: Status 404 returned error can't find the container with id 79da11bb1735ddbffec90dea3537d85f3979213c07f1d8eb03d942694ff06414 Oct 30 18:36:22.719239 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:22.719205 5232 generic.go:356] "Generic (PLEG): container finished" podID="8aee1942-38a1-42af-9826-c316b3363258" containerID="f4b25b24148dc653694ae1625be617c99f6b96725ca027aa14427ca1a3d05b2a" exitCode=0 Oct 30 18:36:22.719650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:22.719298 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp" event={"ID":"8aee1942-38a1-42af-9826-c316b3363258","Type":"ContainerDied","Data":"f4b25b24148dc653694ae1625be617c99f6b96725ca027aa14427ca1a3d05b2a"} Oct 30 18:36:22.719650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:22.719350 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp" event={"ID":"8aee1942-38a1-42af-9826-c316b3363258","Type":"ContainerStarted","Data":"79da11bb1735ddbffec90dea3537d85f3979213c07f1d8eb03d942694ff06414"} Oct 30 18:36:23.856950 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:23.856924 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp" Oct 30 18:36:23.909920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:23.909884 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8aee1942-38a1-42af-9826-c316b3363258-kube-api-access-t9287\" (UniqueName: \"kubernetes.io/projected/8aee1942-38a1-42af-9826-c316b3363258-kube-api-access-t9287\") pod \"8aee1942-38a1-42af-9826-c316b3363258\" (UID: \"8aee1942-38a1-42af-9826-c316b3363258\") " Oct 30 18:36:23.910085 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:23.909930 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8aee1942-38a1-42af-9826-c316b3363258-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aee1942-38a1-42af-9826-c316b3363258-maas-api-xks-ca\") pod \"8aee1942-38a1-42af-9826-c316b3363258\" (UID: \"8aee1942-38a1-42af-9826-c316b3363258\") " Oct 30 18:36:23.910129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:23.910096 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8aee1942-38a1-42af-9826-c316b3363258-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aee1942-38a1-42af-9826-c316b3363258-maas-api-openshift-ca\") pod \"8aee1942-38a1-42af-9826-c316b3363258\" (UID: \"8aee1942-38a1-42af-9826-c316b3363258\") " Oct 30 18:36:23.910441 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:23.910410 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8aee1942-38a1-42af-9826-c316b3363258-maas-api-openshift-ca" pod "8aee1942-38a1-42af-9826-c316b3363258" (UID: "8aee1942-38a1-42af-9826-c316b3363258"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:23.912180 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:23.912157 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8aee1942-38a1-42af-9826-c316b3363258-maas-api-xks-ca" pod "8aee1942-38a1-42af-9826-c316b3363258" (UID: "8aee1942-38a1-42af-9826-c316b3363258"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:23.912659 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:23.912637 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8aee1942-38a1-42af-9826-c316b3363258-kube-api-access-t9287" pod "8aee1942-38a1-42af-9826-c316b3363258" (UID: "8aee1942-38a1-42af-9826-c316b3363258"). InnerVolumeSpecName "kube-api-access-t9287". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:24.011849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:24.011815 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t9287\" (UniqueName: \"kubernetes.io/projected/8aee1942-38a1-42af-9826-c316b3363258-kube-api-access-t9287\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.011849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:24.011845 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8aee1942-38a1-42af-9826-c316b3363258-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.012049 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:24.011859 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8aee1942-38a1-42af-9826-c316b3363258-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:36:24.730313 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:24.730283 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp" Oct 30 18:36:24.730492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:24.730321 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp" event={"ID":"8aee1942-38a1-42af-9826-c316b3363258","Type":"ContainerDied","Data":"79da11bb1735ddbffec90dea3537d85f3979213c07f1d8eb03d942694ff06414"} Oct 30 18:36:24.730492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:24.730368 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79da11bb1735ddbffec90dea3537d85f3979213c07f1d8eb03d942694ff06414" Oct 30 18:36:27.032122 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:36:27.032089 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:36:32.652448 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:32.652412 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq"] Oct 30 18:36:32.652863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:32.652792 5232 cpu_manager.go:414] "containerMap: removing container" podUID="8aee1942-38a1-42af-9826-c316b3363258" containerName="revoke-keys" Oct 30 18:36:32.652863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:32.652804 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8aee1942-38a1-42af-9826-c316b3363258" containerName="revoke-keys" Oct 30 18:36:32.652863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:32.652862 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="8aee1942-38a1-42af-9826-c316b3363258" containerName="revoke-keys" Oct 30 18:36:32.656199 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:32.656178 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq" Oct 30 18:36:32.683279 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:32.683251 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:36:32.683461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:32.683316 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:36:32.686689 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:32.686662 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq"] Oct 30 18:36:32.791238 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:32.791203 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsxwt\" (UniqueName: \"kubernetes.io/projected/ad05b971-8c8d-4dd8-ab52-e402486ca16a-kube-api-access-xsxwt\") pod \"maas-api-revoke-sub-ec843018-w5mqq\" (UID: \"ad05b971-8c8d-4dd8-ab52-e402486ca16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq" Oct 30 18:36:32.791408 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:32.791253 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad05b971-8c8d-4dd8-ab52-e402486ca16a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-w5mqq\" (UID: \"ad05b971-8c8d-4dd8-ab52-e402486ca16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq" Oct 30 18:36:32.791408 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:32.791353 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad05b971-8c8d-4dd8-ab52-e402486ca16a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-w5mqq\" (UID: \"ad05b971-8c8d-4dd8-ab52-e402486ca16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq" Oct 30 18:36:32.892810 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:32.892777 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsxwt\" (UniqueName: \"kubernetes.io/projected/ad05b971-8c8d-4dd8-ab52-e402486ca16a-kube-api-access-xsxwt\") pod \"maas-api-revoke-sub-ec843018-w5mqq\" (UID: \"ad05b971-8c8d-4dd8-ab52-e402486ca16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq" Oct 30 18:36:32.893014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:32.892837 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad05b971-8c8d-4dd8-ab52-e402486ca16a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-w5mqq\" (UID: \"ad05b971-8c8d-4dd8-ab52-e402486ca16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq" Oct 30 18:36:32.893014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:32.892871 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad05b971-8c8d-4dd8-ab52-e402486ca16a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-w5mqq\" (UID: \"ad05b971-8c8d-4dd8-ab52-e402486ca16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq" Oct 30 18:36:32.893442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:32.893424 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad05b971-8c8d-4dd8-ab52-e402486ca16a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-w5mqq\" (UID: \"ad05b971-8c8d-4dd8-ab52-e402486ca16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq" Oct 30 18:36:32.895282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:32.895265 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad05b971-8c8d-4dd8-ab52-e402486ca16a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-w5mqq\" (UID: \"ad05b971-8c8d-4dd8-ab52-e402486ca16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq" Oct 30 18:36:32.913497 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:32.913444 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsxwt\" (UniqueName: \"kubernetes.io/projected/ad05b971-8c8d-4dd8-ab52-e402486ca16a-kube-api-access-xsxwt\") pod \"maas-api-revoke-sub-ec843018-w5mqq\" (UID: \"ad05b971-8c8d-4dd8-ab52-e402486ca16a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq" Oct 30 18:36:32.968211 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:32.968180 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq" Oct 30 18:36:33.262125 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:33.262090 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq"] Oct 30 18:36:33.263054 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:36:33.263025 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad05b971_8c8d_4dd8_ab52_e402486ca16a.slice/crio-0576036d855a30f100f11966dbb88f11d0d527cac3b4202378f62844e6c60525 WatchSource:0}: Error finding container 0576036d855a30f100f11966dbb88f11d0d527cac3b4202378f62844e6c60525: Status 404 returned error can't find the container with id 0576036d855a30f100f11966dbb88f11d0d527cac3b4202378f62844e6c60525 Oct 30 18:36:33.773149 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:33.773058 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq" event={"ID":"ad05b971-8c8d-4dd8-ab52-e402486ca16a","Type":"ContainerStarted","Data":"af60d57f9c152e17172d6de362b6935d9a822a87fcf07b272368cb6951fdd5d4"} Oct 30 18:36:33.773149 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:33.773108 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq" event={"ID":"ad05b971-8c8d-4dd8-ab52-e402486ca16a","Type":"ContainerStarted","Data":"0576036d855a30f100f11966dbb88f11d0d527cac3b4202378f62844e6c60525"} Oct 30 18:36:33.811799 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:33.811754 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq" podStartSLOduration=1.8117386610000001 podStartE2EDuration="1.811738661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:36:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:36:33.81118789 +0000 UTC m=+2093.127608212" watchObservedRunningTime="2026-10-30 18:36:33.811738661 +0000 UTC m=+2093.128159007" Oct 30 18:36:34.778893 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:34.778858 5232 generic.go:356] "Generic (PLEG): container finished" podID="ad05b971-8c8d-4dd8-ab52-e402486ca16a" containerID="af60d57f9c152e17172d6de362b6935d9a822a87fcf07b272368cb6951fdd5d4" exitCode=0 Oct 30 18:36:34.779350 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:34.778919 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq" event={"ID":"ad05b971-8c8d-4dd8-ab52-e402486ca16a","Type":"ContainerDied","Data":"af60d57f9c152e17172d6de362b6935d9a822a87fcf07b272368cb6951fdd5d4"} Oct 30 18:36:35.911465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:35.911442 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq" Oct 30 18:36:36.022431 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:36.022396 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad05b971-8c8d-4dd8-ab52-e402486ca16a-kube-api-access-xsxwt\" (UniqueName: \"kubernetes.io/projected/ad05b971-8c8d-4dd8-ab52-e402486ca16a-kube-api-access-xsxwt\") pod \"ad05b971-8c8d-4dd8-ab52-e402486ca16a\" (UID: \"ad05b971-8c8d-4dd8-ab52-e402486ca16a\") " Oct 30 18:36:36.022654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:36.022442 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ad05b971-8c8d-4dd8-ab52-e402486ca16a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad05b971-8c8d-4dd8-ab52-e402486ca16a-maas-api-xks-ca\") pod \"ad05b971-8c8d-4dd8-ab52-e402486ca16a\" (UID: \"ad05b971-8c8d-4dd8-ab52-e402486ca16a\") " Oct 30 18:36:36.022654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:36.022491 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad05b971-8c8d-4dd8-ab52-e402486ca16a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad05b971-8c8d-4dd8-ab52-e402486ca16a-maas-api-openshift-ca\") pod \"ad05b971-8c8d-4dd8-ab52-e402486ca16a\" (UID: \"ad05b971-8c8d-4dd8-ab52-e402486ca16a\") " Oct 30 18:36:36.022933 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:36.022910 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad05b971-8c8d-4dd8-ab52-e402486ca16a-maas-api-openshift-ca" pod "ad05b971-8c8d-4dd8-ab52-e402486ca16a" (UID: "ad05b971-8c8d-4dd8-ab52-e402486ca16a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:36:36.024654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:36.024629 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad05b971-8c8d-4dd8-ab52-e402486ca16a-kube-api-access-xsxwt" pod "ad05b971-8c8d-4dd8-ab52-e402486ca16a" (UID: "ad05b971-8c8d-4dd8-ab52-e402486ca16a"). InnerVolumeSpecName "kube-api-access-xsxwt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:36:36.024654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:36.024641 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ad05b971-8c8d-4dd8-ab52-e402486ca16a-maas-api-xks-ca" pod "ad05b971-8c8d-4dd8-ab52-e402486ca16a" (UID: "ad05b971-8c8d-4dd8-ab52-e402486ca16a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:36:36.124048 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:36.123969 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsxwt\" (UniqueName: \"kubernetes.io/projected/ad05b971-8c8d-4dd8-ab52-e402486ca16a-kube-api-access-xsxwt\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:36:36.124048 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:36.123999 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ad05b971-8c8d-4dd8-ab52-e402486ca16a-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:36:36.124048 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:36.124008 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ad05b971-8c8d-4dd8-ab52-e402486ca16a-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:36:36.792451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:36.792414 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq" Oct 30 18:36:36.792693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:36.792415 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq" event={"ID":"ad05b971-8c8d-4dd8-ab52-e402486ca16a","Type":"ContainerDied","Data":"0576036d855a30f100f11966dbb88f11d0d527cac3b4202378f62844e6c60525"} Oct 30 18:36:36.792693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:36.792565 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0576036d855a30f100f11966dbb88f11d0d527cac3b4202378f62844e6c60525" Oct 30 18:36:46.025553 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:46.025493 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd"] Oct 30 18:36:46.037586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:46.037562 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8f12b025-rp8gd"] Oct 30 18:36:46.800425 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:46.800391 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="31ff171d-ddfa-4212-b9eb-8c47d790a3ec" path="/var/lib/kubelet/pods/31ff171d-ddfa-4212-b9eb-8c47d790a3ec/volumes" Oct 30 18:36:54.024534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:54.024480 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp"] Oct 30 18:36:54.036637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:54.036610 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-8b574ab3-x4kkp"] Oct 30 18:36:54.801524 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:54.801474 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07405f12-8a6c-4a2d-af64-70650200a1cf" path="/var/lib/kubelet/pods/07405f12-8a6c-4a2d-af64-70650200a1cf/volumes" Oct 30 18:36:55.374407 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:55.374375 5232 scope.go:122] "RemoveContainer" containerID="d9e1f544f3a0e7de1723c765c08627973d4e4e0e245ed3c5054326a383218caa" Oct 30 18:36:57.033186 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:36:57.033140 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:36:58.022956 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:58.022926 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82"] Oct 30 18:36:58.034877 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:58.034850 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ad1ec265-5zp82"] Oct 30 18:36:58.801115 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:36:58.801079 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f06bdf8-de96-4dce-8971-810550144958" path="/var/lib/kubelet/pods/5f06bdf8-de96-4dce-8971-810550144958/volumes" Oct 30 18:37:01.561194 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:01.561157 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h"] Oct 30 18:37:01.561665 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:01.561610 5232 cpu_manager.go:414] "containerMap: removing container" podUID="ad05b971-8c8d-4dd8-ab52-e402486ca16a" containerName="revoke-keys" Oct 30 18:37:01.561665 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:01.561624 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad05b971-8c8d-4dd8-ab52-e402486ca16a" containerName="revoke-keys" Oct 30 18:37:01.561755 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:01.561681 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad05b971-8c8d-4dd8-ab52-e402486ca16a" containerName="revoke-keys" Oct 30 18:37:01.564639 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:01.564620 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h" Oct 30 18:37:01.582865 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:01.582843 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h"] Oct 30 18:37:01.592723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:01.592695 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:37:01.592850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:01.592727 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:37:01.643285 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:01.643249 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xhh7h\" (UID: \"eaf5cf88-7d58-49a4-8dba-e56ede8eb25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h" Oct 30 18:37:01.643285 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:01.643289 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wc8rf\" (UniqueName: \"kubernetes.io/projected/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-kube-api-access-wc8rf\") pod \"maas-api-revoke-sub-046e1343-xhh7h\" (UID: \"eaf5cf88-7d58-49a4-8dba-e56ede8eb25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h" Oct 30 18:37:01.643496 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:01.643382 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xhh7h\" (UID: \"eaf5cf88-7d58-49a4-8dba-e56ede8eb25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h" Oct 30 18:37:01.744638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:01.744602 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xhh7h\" (UID: \"eaf5cf88-7d58-49a4-8dba-e56ede8eb25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h" Oct 30 18:37:01.744638 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:01.744644 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wc8rf\" (UniqueName: \"kubernetes.io/projected/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-kube-api-access-wc8rf\") pod \"maas-api-revoke-sub-046e1343-xhh7h\" (UID: \"eaf5cf88-7d58-49a4-8dba-e56ede8eb25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h" Oct 30 18:37:01.744848 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:01.744673 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xhh7h\" (UID: \"eaf5cf88-7d58-49a4-8dba-e56ede8eb25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h" Oct 30 18:37:01.745246 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:01.745218 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-xhh7h\" (UID: \"eaf5cf88-7d58-49a4-8dba-e56ede8eb25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h" Oct 30 18:37:01.747121 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:01.747102 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-xhh7h\" (UID: \"eaf5cf88-7d58-49a4-8dba-e56ede8eb25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h" Oct 30 18:37:01.764582 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:01.764557 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wc8rf\" (UniqueName: \"kubernetes.io/projected/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-kube-api-access-wc8rf\") pod \"maas-api-revoke-sub-046e1343-xhh7h\" (UID: \"eaf5cf88-7d58-49a4-8dba-e56ede8eb25e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h" Oct 30 18:37:01.877323 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:01.877259 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h" Oct 30 18:37:02.168140 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:02.168116 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h"] Oct 30 18:37:02.170573 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:37:02.170544 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeaf5cf88_7d58_49a4_8dba_e56ede8eb25e.slice/crio-1c72c46da4821a4715cd36b9bc87ffc7640fa2439d829edd4920076ab80ce014 WatchSource:0}: Error finding container 1c72c46da4821a4715cd36b9bc87ffc7640fa2439d829edd4920076ab80ce014: Status 404 returned error can't find the container with id 1c72c46da4821a4715cd36b9bc87ffc7640fa2439d829edd4920076ab80ce014 Oct 30 18:37:02.952538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:02.952487 5232 generic.go:356] "Generic (PLEG): container finished" podID="eaf5cf88-7d58-49a4-8dba-e56ede8eb25e" containerID="cde04c97451bacc0bbc6aeb071c3d41e723953e54a6da7a0452a18791bf23110" exitCode=0 Oct 30 18:37:02.952906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:02.952584 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h" event={"ID":"eaf5cf88-7d58-49a4-8dba-e56ede8eb25e","Type":"ContainerDied","Data":"cde04c97451bacc0bbc6aeb071c3d41e723953e54a6da7a0452a18791bf23110"} Oct 30 18:37:02.952906 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:02.952644 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h" event={"ID":"eaf5cf88-7d58-49a4-8dba-e56ede8eb25e","Type":"ContainerStarted","Data":"1c72c46da4821a4715cd36b9bc87ffc7640fa2439d829edd4920076ab80ce014"} Oct 30 18:37:04.082326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:04.082300 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h" Oct 30 18:37:04.166523 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:04.166487 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-kube-api-access-wc8rf\" (UniqueName: \"kubernetes.io/projected/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-kube-api-access-wc8rf\") pod \"eaf5cf88-7d58-49a4-8dba-e56ede8eb25e\" (UID: \"eaf5cf88-7d58-49a4-8dba-e56ede8eb25e\") " Oct 30 18:37:04.166713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:04.166546 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-maas-api-xks-ca\") pod \"eaf5cf88-7d58-49a4-8dba-e56ede8eb25e\" (UID: \"eaf5cf88-7d58-49a4-8dba-e56ede8eb25e\") " Oct 30 18:37:04.166713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:04.166573 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-maas-api-openshift-ca\") pod \"eaf5cf88-7d58-49a4-8dba-e56ede8eb25e\" (UID: \"eaf5cf88-7d58-49a4-8dba-e56ede8eb25e\") " Oct 30 18:37:04.166999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:04.166974 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-maas-api-openshift-ca" pod "eaf5cf88-7d58-49a4-8dba-e56ede8eb25e" (UID: "eaf5cf88-7d58-49a4-8dba-e56ede8eb25e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:04.168646 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:04.168626 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-kube-api-access-wc8rf" pod "eaf5cf88-7d58-49a4-8dba-e56ede8eb25e" (UID: "eaf5cf88-7d58-49a4-8dba-e56ede8eb25e"). InnerVolumeSpecName "kube-api-access-wc8rf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:04.168866 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:04.168853 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-maas-api-xks-ca" pod "eaf5cf88-7d58-49a4-8dba-e56ede8eb25e" (UID: "eaf5cf88-7d58-49a4-8dba-e56ede8eb25e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:04.267441 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:04.267405 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wc8rf\" (UniqueName: \"kubernetes.io/projected/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-kube-api-access-wc8rf\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:37:04.267441 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:04.267437 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:37:04.267726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:04.267455 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:37:04.963892 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:04.963864 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h" Oct 30 18:37:04.964059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:04.963862 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h" event={"ID":"eaf5cf88-7d58-49a4-8dba-e56ede8eb25e","Type":"ContainerDied","Data":"1c72c46da4821a4715cd36b9bc87ffc7640fa2439d829edd4920076ab80ce014"} Oct 30 18:37:04.964059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:04.963989 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c72c46da4821a4715cd36b9bc87ffc7640fa2439d829edd4920076ab80ce014" Oct 30 18:37:11.031650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:11.031608 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb"] Oct 30 18:37:11.045093 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:11.045058 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-42b56365-tf5kb"] Oct 30 18:37:11.895319 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:11.895285 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb"] Oct 30 18:37:11.895673 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:11.895660 5232 cpu_manager.go:414] "containerMap: removing container" podUID="eaf5cf88-7d58-49a4-8dba-e56ede8eb25e" containerName="revoke-keys" Oct 30 18:37:11.895737 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:11.895674 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eaf5cf88-7d58-49a4-8dba-e56ede8eb25e" containerName="revoke-keys" Oct 30 18:37:11.895737 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:11.895735 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="eaf5cf88-7d58-49a4-8dba-e56ede8eb25e" containerName="revoke-keys" Oct 30 18:37:11.898821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:11.898802 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb" Oct 30 18:37:11.925659 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:11.925631 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:37:11.925831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:11.925792 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:37:11.928466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:11.928443 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb"] Oct 30 18:37:12.033121 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:12.033077 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qcbnl\" (UniqueName: \"kubernetes.io/projected/5fac5b90-e6ca-4a1f-9014-61149123cc5f-kube-api-access-qcbnl\") pod \"maas-api-revoke-sub-d3fba056-rrzmb\" (UID: \"5fac5b90-e6ca-4a1f-9014-61149123cc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb" Oct 30 18:37:12.033619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:12.033129 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fac5b90-e6ca-4a1f-9014-61149123cc5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rrzmb\" (UID: \"5fac5b90-e6ca-4a1f-9014-61149123cc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb" Oct 30 18:37:12.033619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:12.033271 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fac5b90-e6ca-4a1f-9014-61149123cc5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rrzmb\" (UID: \"5fac5b90-e6ca-4a1f-9014-61149123cc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb" Oct 30 18:37:12.133819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:12.133772 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fac5b90-e6ca-4a1f-9014-61149123cc5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rrzmb\" (UID: \"5fac5b90-e6ca-4a1f-9014-61149123cc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb" Oct 30 18:37:12.134011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:12.133843 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qcbnl\" (UniqueName: \"kubernetes.io/projected/5fac5b90-e6ca-4a1f-9014-61149123cc5f-kube-api-access-qcbnl\") pod \"maas-api-revoke-sub-d3fba056-rrzmb\" (UID: \"5fac5b90-e6ca-4a1f-9014-61149123cc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb" Oct 30 18:37:12.134011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:12.133878 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fac5b90-e6ca-4a1f-9014-61149123cc5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rrzmb\" (UID: \"5fac5b90-e6ca-4a1f-9014-61149123cc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb" Oct 30 18:37:12.134468 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:12.134448 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fac5b90-e6ca-4a1f-9014-61149123cc5f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-rrzmb\" (UID: \"5fac5b90-e6ca-4a1f-9014-61149123cc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb" Oct 30 18:37:12.136439 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:12.136417 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fac5b90-e6ca-4a1f-9014-61149123cc5f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-rrzmb\" (UID: \"5fac5b90-e6ca-4a1f-9014-61149123cc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb" Oct 30 18:37:12.160174 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:12.160110 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qcbnl\" (UniqueName: \"kubernetes.io/projected/5fac5b90-e6ca-4a1f-9014-61149123cc5f-kube-api-access-qcbnl\") pod \"maas-api-revoke-sub-d3fba056-rrzmb\" (UID: \"5fac5b90-e6ca-4a1f-9014-61149123cc5f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb" Oct 30 18:37:12.210903 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:12.210875 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb" Oct 30 18:37:12.499477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:12.499447 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb"] Oct 30 18:37:12.501294 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:37:12.501255 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5fac5b90_e6ca_4a1f_9014_61149123cc5f.slice/crio-0b2913738ad3f28cb96e66109fad9214b2236656d1ef13bd974b0732b0f38144 WatchSource:0}: Error finding container 0b2913738ad3f28cb96e66109fad9214b2236656d1ef13bd974b0732b0f38144: Status 404 returned error can't find the container with id 0b2913738ad3f28cb96e66109fad9214b2236656d1ef13bd974b0732b0f38144 Oct 30 18:37:12.802107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:12.802032 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62472024-eb2f-418e-a074-acb6bf6b46c6" path="/var/lib/kubelet/pods/62472024-eb2f-418e-a074-acb6bf6b46c6/volumes" Oct 30 18:37:13.004806 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:13.004770 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb" event={"ID":"5fac5b90-e6ca-4a1f-9014-61149123cc5f","Type":"ContainerStarted","Data":"dcb3cf9a20af9a2f28f7afd08377e1924e24309ce48c662fad8137d195a34002"} Oct 30 18:37:13.004991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:13.004828 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb" event={"ID":"5fac5b90-e6ca-4a1f-9014-61149123cc5f","Type":"ContainerStarted","Data":"0b2913738ad3f28cb96e66109fad9214b2236656d1ef13bd974b0732b0f38144"} Oct 30 18:37:13.043572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:13.043522 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb" podStartSLOduration=2.043491653 podStartE2EDuration="2.043491653s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:37:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:37:13.042100836 +0000 UTC m=+2132.358521157" watchObservedRunningTime="2026-10-30 18:37:13.043491653 +0000 UTC m=+2132.359911976" Oct 30 18:37:14.014156 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:14.014111 5232 generic.go:356] "Generic (PLEG): container finished" podID="5fac5b90-e6ca-4a1f-9014-61149123cc5f" containerID="dcb3cf9a20af9a2f28f7afd08377e1924e24309ce48c662fad8137d195a34002" exitCode=0 Oct 30 18:37:14.014387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:14.014182 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb" event={"ID":"5fac5b90-e6ca-4a1f-9014-61149123cc5f","Type":"ContainerDied","Data":"dcb3cf9a20af9a2f28f7afd08377e1924e24309ce48c662fad8137d195a34002"} Oct 30 18:37:15.821230 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:15.820892 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb" Oct 30 18:37:15.828553 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:15.826597 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zzlvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:37:15.828553 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:15.826659 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" podUID="b075648b-215b-4d0e-b3dc-41ea5ed46ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:37:16.007837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:16.007801 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5fac5b90-e6ca-4a1f-9014-61149123cc5f-kube-api-access-qcbnl\" (UniqueName: \"kubernetes.io/projected/5fac5b90-e6ca-4a1f-9014-61149123cc5f-kube-api-access-qcbnl\") pod \"5fac5b90-e6ca-4a1f-9014-61149123cc5f\" (UID: \"5fac5b90-e6ca-4a1f-9014-61149123cc5f\") " Oct 30 18:37:16.008206 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:16.008182 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5fac5b90-e6ca-4a1f-9014-61149123cc5f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fac5b90-e6ca-4a1f-9014-61149123cc5f-maas-api-openshift-ca\") pod \"5fac5b90-e6ca-4a1f-9014-61149123cc5f\" (UID: \"5fac5b90-e6ca-4a1f-9014-61149123cc5f\") " Oct 30 18:37:16.008378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:16.008359 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5fac5b90-e6ca-4a1f-9014-61149123cc5f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fac5b90-e6ca-4a1f-9014-61149123cc5f-maas-api-xks-ca\") pod \"5fac5b90-e6ca-4a1f-9014-61149123cc5f\" (UID: \"5fac5b90-e6ca-4a1f-9014-61149123cc5f\") " Oct 30 18:37:16.036022 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:16.035985 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5fac5b90-e6ca-4a1f-9014-61149123cc5f-maas-api-xks-ca" pod "5fac5b90-e6ca-4a1f-9014-61149123cc5f" (UID: "5fac5b90-e6ca-4a1f-9014-61149123cc5f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:16.054718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:16.037591 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kbsdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:37:16.054718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:16.037648 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" podUID="92403b9c-a622-457e-8b9b-a92f86f93b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:37:16.054718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:16.041701 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5fac5b90-e6ca-4a1f-9014-61149123cc5f-maas-api-openshift-ca" pod "5fac5b90-e6ca-4a1f-9014-61149123cc5f" (UID: "5fac5b90-e6ca-4a1f-9014-61149123cc5f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:16.063534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:16.057948 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5fac5b90-e6ca-4a1f-9014-61149123cc5f-kube-api-access-qcbnl" pod "5fac5b90-e6ca-4a1f-9014-61149123cc5f" (UID: "5fac5b90-e6ca-4a1f-9014-61149123cc5f"). InnerVolumeSpecName "kube-api-access-qcbnl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:16.096537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:16.095473 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb" event={"ID":"5fac5b90-e6ca-4a1f-9014-61149123cc5f","Type":"ContainerDied","Data":"0b2913738ad3f28cb96e66109fad9214b2236656d1ef13bd974b0732b0f38144"} Oct 30 18:37:16.096958 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:16.096938 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0b2913738ad3f28cb96e66109fad9214b2236656d1ef13bd974b0732b0f38144" Oct 30 18:37:16.097158 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:16.096814 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb" Oct 30 18:37:16.114205 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:16.114130 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5fac5b90-e6ca-4a1f-9014-61149123cc5f-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:37:16.114205 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:16.114164 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5fac5b90-e6ca-4a1f-9014-61149123cc5f-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:37:16.114205 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:16.114180 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qcbnl\" (UniqueName: \"kubernetes.io/projected/5fac5b90-e6ca-4a1f-9014-61149123cc5f-kube-api-access-qcbnl\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:37:27.033177 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:37:27.033131 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:37:30.826720 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:30.826680 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zzlvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:37:30.827365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:30.827329 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" podUID="b075648b-215b-4d0e-b3dc-41ea5ed46ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:37:31.038533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:31.037622 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kbsdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:37:31.038533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:31.037693 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" podUID="92403b9c-a622-457e-8b9b-a92f86f93b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:37:37.386219 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.386175 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5"] Oct 30 18:37:37.386793 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.386771 5232 cpu_manager.go:414] "containerMap: removing container" podUID="5fac5b90-e6ca-4a1f-9014-61149123cc5f" containerName="revoke-keys" Oct 30 18:37:37.386886 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.386795 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5fac5b90-e6ca-4a1f-9014-61149123cc5f" containerName="revoke-keys" Oct 30 18:37:37.386886 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.386882 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="5fac5b90-e6ca-4a1f-9014-61149123cc5f" containerName="revoke-keys" Oct 30 18:37:37.389949 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.389927 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5" Oct 30 18:37:37.408243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.408218 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5"] Oct 30 18:37:37.417942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.417914 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:37:37.418096 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.417959 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:37:37.441141 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.441115 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcdcc782-a4b3-455e-b95e-fe3d5011093f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mmnn5\" (UID: \"fcdcc782-a4b3-455e-b95e-fe3d5011093f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5" Oct 30 18:37:37.441306 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.441154 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xp5j9\" (UniqueName: \"kubernetes.io/projected/fcdcc782-a4b3-455e-b95e-fe3d5011093f-kube-api-access-xp5j9\") pod \"maas-api-revoke-sub-ae594979-mmnn5\" (UID: \"fcdcc782-a4b3-455e-b95e-fe3d5011093f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5" Oct 30 18:37:37.441306 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.441270 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcdcc782-a4b3-455e-b95e-fe3d5011093f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mmnn5\" (UID: \"fcdcc782-a4b3-455e-b95e-fe3d5011093f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5" Oct 30 18:37:37.542480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.542448 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcdcc782-a4b3-455e-b95e-fe3d5011093f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mmnn5\" (UID: \"fcdcc782-a4b3-455e-b95e-fe3d5011093f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5" Oct 30 18:37:37.542668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.542549 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcdcc782-a4b3-455e-b95e-fe3d5011093f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mmnn5\" (UID: \"fcdcc782-a4b3-455e-b95e-fe3d5011093f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5" Oct 30 18:37:37.542668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.542586 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xp5j9\" (UniqueName: \"kubernetes.io/projected/fcdcc782-a4b3-455e-b95e-fe3d5011093f-kube-api-access-xp5j9\") pod \"maas-api-revoke-sub-ae594979-mmnn5\" (UID: \"fcdcc782-a4b3-455e-b95e-fe3d5011093f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5" Oct 30 18:37:37.543244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.543183 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcdcc782-a4b3-455e-b95e-fe3d5011093f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-mmnn5\" (UID: \"fcdcc782-a4b3-455e-b95e-fe3d5011093f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5" Oct 30 18:37:37.544976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.544952 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcdcc782-a4b3-455e-b95e-fe3d5011093f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-mmnn5\" (UID: \"fcdcc782-a4b3-455e-b95e-fe3d5011093f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5" Oct 30 18:37:37.562414 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.562387 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xp5j9\" (UniqueName: \"kubernetes.io/projected/fcdcc782-a4b3-455e-b95e-fe3d5011093f-kube-api-access-xp5j9\") pod \"maas-api-revoke-sub-ae594979-mmnn5\" (UID: \"fcdcc782-a4b3-455e-b95e-fe3d5011093f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5" Oct 30 18:37:37.703047 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.702973 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5" Oct 30 18:37:37.991247 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:37.991124 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5"] Oct 30 18:37:37.993438 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:37:37.993412 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcdcc782_a4b3_455e_b95e_fe3d5011093f.slice/crio-a34f0b7ae8c67c5ec4422a03d316759c9905e299e6a1cdd4747d8812995502b1 WatchSource:0}: Error finding container a34f0b7ae8c67c5ec4422a03d316759c9905e299e6a1cdd4747d8812995502b1: Status 404 returned error can't find the container with id a34f0b7ae8c67c5ec4422a03d316759c9905e299e6a1cdd4747d8812995502b1 Oct 30 18:37:38.232165 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:38.232134 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5" event={"ID":"fcdcc782-a4b3-455e-b95e-fe3d5011093f","Type":"ContainerStarted","Data":"a34f0b7ae8c67c5ec4422a03d316759c9905e299e6a1cdd4747d8812995502b1"} Oct 30 18:37:39.238195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:39.238165 5232 generic.go:356] "Generic (PLEG): container finished" podID="fcdcc782-a4b3-455e-b95e-fe3d5011093f" containerID="89d76d33557966a5d8352082279b5ccfb72f74fceeb3a67256155b227f92cf99" exitCode=0 Oct 30 18:37:39.238631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:39.238262 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5" event={"ID":"fcdcc782-a4b3-455e-b95e-fe3d5011093f","Type":"ContainerDied","Data":"89d76d33557966a5d8352082279b5ccfb72f74fceeb3a67256155b227f92cf99"} Oct 30 18:37:40.376744 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:40.376716 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5" Oct 30 18:37:40.470344 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:40.470313 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fcdcc782-a4b3-455e-b95e-fe3d5011093f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcdcc782-a4b3-455e-b95e-fe3d5011093f-maas-api-xks-ca\") pod \"fcdcc782-a4b3-455e-b95e-fe3d5011093f\" (UID: \"fcdcc782-a4b3-455e-b95e-fe3d5011093f\") " Oct 30 18:37:40.470538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:40.470354 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fcdcc782-a4b3-455e-b95e-fe3d5011093f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcdcc782-a4b3-455e-b95e-fe3d5011093f-maas-api-openshift-ca\") pod \"fcdcc782-a4b3-455e-b95e-fe3d5011093f\" (UID: \"fcdcc782-a4b3-455e-b95e-fe3d5011093f\") " Oct 30 18:37:40.470538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:40.470389 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fcdcc782-a4b3-455e-b95e-fe3d5011093f-kube-api-access-xp5j9\" (UniqueName: \"kubernetes.io/projected/fcdcc782-a4b3-455e-b95e-fe3d5011093f-kube-api-access-xp5j9\") pod \"fcdcc782-a4b3-455e-b95e-fe3d5011093f\" (UID: \"fcdcc782-a4b3-455e-b95e-fe3d5011093f\") " Oct 30 18:37:40.470832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:40.470804 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fcdcc782-a4b3-455e-b95e-fe3d5011093f-maas-api-openshift-ca" pod "fcdcc782-a4b3-455e-b95e-fe3d5011093f" (UID: "fcdcc782-a4b3-455e-b95e-fe3d5011093f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:37:40.472614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:40.472592 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fcdcc782-a4b3-455e-b95e-fe3d5011093f-kube-api-access-xp5j9" pod "fcdcc782-a4b3-455e-b95e-fe3d5011093f" (UID: "fcdcc782-a4b3-455e-b95e-fe3d5011093f"). InnerVolumeSpecName "kube-api-access-xp5j9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:37:40.472699 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:40.472621 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fcdcc782-a4b3-455e-b95e-fe3d5011093f-maas-api-xks-ca" pod "fcdcc782-a4b3-455e-b95e-fe3d5011093f" (UID: "fcdcc782-a4b3-455e-b95e-fe3d5011093f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:37:40.571565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:40.571487 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fcdcc782-a4b3-455e-b95e-fe3d5011093f-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:37:40.571565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:40.571528 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fcdcc782-a4b3-455e-b95e-fe3d5011093f-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:37:40.571565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:40.571538 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xp5j9\" (UniqueName: \"kubernetes.io/projected/fcdcc782-a4b3-455e-b95e-fe3d5011093f-kube-api-access-xp5j9\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:37:41.249930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:41.249887 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5" event={"ID":"fcdcc782-a4b3-455e-b95e-fe3d5011093f","Type":"ContainerDied","Data":"a34f0b7ae8c67c5ec4422a03d316759c9905e299e6a1cdd4747d8812995502b1"} Oct 30 18:37:41.249930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:41.249911 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5" Oct 30 18:37:41.250143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:41.249938 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a34f0b7ae8c67c5ec4422a03d316759c9905e299e6a1cdd4747d8812995502b1" Oct 30 18:37:55.396411 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:55.396304 5232 scope.go:122] "RemoveContainer" containerID="6bebe22962bdfeb4f3f3d0cf99fb4e5d3c37e56da15c9241cd31472c59a7613a" Oct 30 18:37:55.409315 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:55.409293 5232 scope.go:122] "RemoveContainer" containerID="09cd5f64eb7617cf8ec190ced5ed13edccc686451ca466c8098c3f97a8b038a6" Oct 30 18:37:55.422494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:37:55.422370 5232 scope.go:122] "RemoveContainer" containerID="7e79a8b167ba9698c0776c0cb58696990c3cd2d13192427f88a981e6e4a8dbcb" Oct 30 18:37:57.032816 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:37:57.032779 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:38:02.438033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:02.437999 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp"] Oct 30 18:38:02.438649 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:02.438622 5232 cpu_manager.go:414] "containerMap: removing container" podUID="fcdcc782-a4b3-455e-b95e-fe3d5011093f" containerName="revoke-keys" Oct 30 18:38:02.438649 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:02.438648 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fcdcc782-a4b3-455e-b95e-fe3d5011093f" containerName="revoke-keys" Oct 30 18:38:02.438837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:02.438753 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="fcdcc782-a4b3-455e-b95e-fe3d5011093f" containerName="revoke-keys" Oct 30 18:38:02.443249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:02.443230 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp" Oct 30 18:38:02.459128 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:02.459099 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp"] Oct 30 18:38:02.469064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:02.469037 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:38:02.469215 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:02.469104 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:38:02.609931 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:02.609898 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bqvlz\" (UniqueName: \"kubernetes.io/projected/e13ef4dd-0358-469c-a655-d447f298aec2-kube-api-access-bqvlz\") pod \"maas-api-revoke-sub-a4aef7d7-7frnp\" (UID: \"e13ef4dd-0358-469c-a655-d447f298aec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp" Oct 30 18:38:02.610114 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:02.609944 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13ef4dd-0358-469c-a655-d447f298aec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7frnp\" (UID: \"e13ef4dd-0358-469c-a655-d447f298aec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp" Oct 30 18:38:02.610114 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:02.610026 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13ef4dd-0358-469c-a655-d447f298aec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7frnp\" (UID: \"e13ef4dd-0358-469c-a655-d447f298aec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp" Oct 30 18:38:02.711543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:02.711430 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13ef4dd-0358-469c-a655-d447f298aec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7frnp\" (UID: \"e13ef4dd-0358-469c-a655-d447f298aec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp" Oct 30 18:38:02.711543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:02.711495 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bqvlz\" (UniqueName: \"kubernetes.io/projected/e13ef4dd-0358-469c-a655-d447f298aec2-kube-api-access-bqvlz\") pod \"maas-api-revoke-sub-a4aef7d7-7frnp\" (UID: \"e13ef4dd-0358-469c-a655-d447f298aec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp" Oct 30 18:38:02.711749 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:02.711575 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13ef4dd-0358-469c-a655-d447f298aec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7frnp\" (UID: \"e13ef4dd-0358-469c-a655-d447f298aec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp" Oct 30 18:38:02.712240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:02.712197 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13ef4dd-0358-469c-a655-d447f298aec2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7frnp\" (UID: \"e13ef4dd-0358-469c-a655-d447f298aec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp" Oct 30 18:38:02.714107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:02.714080 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13ef4dd-0358-469c-a655-d447f298aec2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-7frnp\" (UID: \"e13ef4dd-0358-469c-a655-d447f298aec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp" Oct 30 18:38:02.732057 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:02.732033 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bqvlz\" (UniqueName: \"kubernetes.io/projected/e13ef4dd-0358-469c-a655-d447f298aec2-kube-api-access-bqvlz\") pod \"maas-api-revoke-sub-a4aef7d7-7frnp\" (UID: \"e13ef4dd-0358-469c-a655-d447f298aec2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp" Oct 30 18:38:02.758112 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:02.758088 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp" Oct 30 18:38:03.053468 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:03.053439 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp"] Oct 30 18:38:03.055417 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:38:03.055388 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode13ef4dd_0358_469c_a655_d447f298aec2.slice/crio-f83b488204b6f5907b04aa7ffed5e389f23eb79bd9db5b49103d3f3922fb3a87 WatchSource:0}: Error finding container f83b488204b6f5907b04aa7ffed5e389f23eb79bd9db5b49103d3f3922fb3a87: Status 404 returned error can't find the container with id f83b488204b6f5907b04aa7ffed5e389f23eb79bd9db5b49103d3f3922fb3a87 Oct 30 18:38:03.383041 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:03.382997 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp" event={"ID":"e13ef4dd-0358-469c-a655-d447f298aec2","Type":"ContainerStarted","Data":"c4409d9e541bc3de5b8d197e1f0e0d968174d1bcb4408601f3649dcddebea997"} Oct 30 18:38:03.383202 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:03.383063 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp" event={"ID":"e13ef4dd-0358-469c-a655-d447f298aec2","Type":"ContainerStarted","Data":"f83b488204b6f5907b04aa7ffed5e389f23eb79bd9db5b49103d3f3922fb3a87"} Oct 30 18:38:03.421158 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:03.421109 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp" podStartSLOduration=1.421096049 podStartE2EDuration="1.421096049s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:03.420548399 +0000 UTC m=+2182.736968719" watchObservedRunningTime="2026-10-30 18:38:03.421096049 +0000 UTC m=+2182.737516370" Oct 30 18:38:04.390525 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:04.390473 5232 generic.go:356] "Generic (PLEG): container finished" podID="e13ef4dd-0358-469c-a655-d447f298aec2" containerID="c4409d9e541bc3de5b8d197e1f0e0d968174d1bcb4408601f3649dcddebea997" exitCode=0 Oct 30 18:38:04.390936 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:04.390561 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp" event={"ID":"e13ef4dd-0358-469c-a655-d447f298aec2","Type":"ContainerDied","Data":"c4409d9e541bc3de5b8d197e1f0e0d968174d1bcb4408601f3649dcddebea997"} Oct 30 18:38:05.523453 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:05.523428 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp" Oct 30 18:38:05.644761 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:05.644724 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e13ef4dd-0358-469c-a655-d447f298aec2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13ef4dd-0358-469c-a655-d447f298aec2-maas-api-xks-ca\") pod \"e13ef4dd-0358-469c-a655-d447f298aec2\" (UID: \"e13ef4dd-0358-469c-a655-d447f298aec2\") " Oct 30 18:38:05.644936 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:05.644839 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e13ef4dd-0358-469c-a655-d447f298aec2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13ef4dd-0358-469c-a655-d447f298aec2-maas-api-openshift-ca\") pod \"e13ef4dd-0358-469c-a655-d447f298aec2\" (UID: \"e13ef4dd-0358-469c-a655-d447f298aec2\") " Oct 30 18:38:05.644936 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:05.644883 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e13ef4dd-0358-469c-a655-d447f298aec2-kube-api-access-bqvlz\" (UniqueName: \"kubernetes.io/projected/e13ef4dd-0358-469c-a655-d447f298aec2-kube-api-access-bqvlz\") pod \"e13ef4dd-0358-469c-a655-d447f298aec2\" (UID: \"e13ef4dd-0358-469c-a655-d447f298aec2\") " Oct 30 18:38:05.645216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:05.645191 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e13ef4dd-0358-469c-a655-d447f298aec2-maas-api-openshift-ca" pod "e13ef4dd-0358-469c-a655-d447f298aec2" (UID: "e13ef4dd-0358-469c-a655-d447f298aec2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:05.647171 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:05.647107 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e13ef4dd-0358-469c-a655-d447f298aec2-maas-api-xks-ca" pod "e13ef4dd-0358-469c-a655-d447f298aec2" (UID: "e13ef4dd-0358-469c-a655-d447f298aec2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:05.647171 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:05.647109 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e13ef4dd-0358-469c-a655-d447f298aec2-kube-api-access-bqvlz" pod "e13ef4dd-0358-469c-a655-d447f298aec2" (UID: "e13ef4dd-0358-469c-a655-d447f298aec2"). InnerVolumeSpecName "kube-api-access-bqvlz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:05.746090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:05.746057 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e13ef4dd-0358-469c-a655-d447f298aec2-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:38:05.746090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:05.746086 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e13ef4dd-0358-469c-a655-d447f298aec2-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:38:05.746090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:05.746096 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bqvlz\" (UniqueName: \"kubernetes.io/projected/e13ef4dd-0358-469c-a655-d447f298aec2-kube-api-access-bqvlz\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:38:06.402786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:06.402756 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp" Oct 30 18:38:06.402786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:06.402764 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp" event={"ID":"e13ef4dd-0358-469c-a655-d447f298aec2","Type":"ContainerDied","Data":"f83b488204b6f5907b04aa7ffed5e389f23eb79bd9db5b49103d3f3922fb3a87"} Oct 30 18:38:06.403009 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:06.402811 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f83b488204b6f5907b04aa7ffed5e389f23eb79bd9db5b49103d3f3922fb3a87" Oct 30 18:38:26.026325 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.026288 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5"] Oct 30 18:38:26.038365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.038332 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7"] Oct 30 18:38:26.039946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.039923 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6e816d55-6sxz5"] Oct 30 18:38:26.043962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.043934 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-7286092f-hs2q7"] Oct 30 18:38:26.800669 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.800634 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="317b89c4-eac3-48eb-8587-a3911c5dd435" path="/var/lib/kubelet/pods/317b89c4-eac3-48eb-8587-a3911c5dd435/volumes" Oct 30 18:38:26.801040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.801027 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d46be40b-6bc7-4633-8389-f7a4d79c603d" path="/var/lib/kubelet/pods/d46be40b-6bc7-4633-8389-f7a4d79c603d/volumes" Oct 30 18:38:26.829899 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.829865 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n"] Oct 30 18:38:26.830225 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.830213 5232 cpu_manager.go:414] "containerMap: removing container" podUID="e13ef4dd-0358-469c-a655-d447f298aec2" containerName="revoke-keys" Oct 30 18:38:26.830274 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.830227 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e13ef4dd-0358-469c-a655-d447f298aec2" containerName="revoke-keys" Oct 30 18:38:26.830311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.830287 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="e13ef4dd-0358-469c-a655-d447f298aec2" containerName="revoke-keys" Oct 30 18:38:26.834560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.834543 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n" Oct 30 18:38:26.841849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.841819 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n"] Oct 30 18:38:26.849303 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.849280 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:38:26.849414 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.849337 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:38:26.859180 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.859150 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2vkp\" (UniqueName: \"kubernetes.io/projected/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-kube-api-access-z2vkp\") pod \"maas-api-revoke-sub-79805ca9-r6w7n\" (UID: \"6ef6f72d-739e-4f95-83de-46f03c3dbaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n" Oct 30 18:38:26.859313 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.859190 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-r6w7n\" (UID: \"6ef6f72d-739e-4f95-83de-46f03c3dbaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n" Oct 30 18:38:26.859387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.859359 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-r6w7n\" (UID: \"6ef6f72d-739e-4f95-83de-46f03c3dbaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n" Oct 30 18:38:26.960543 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.960490 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-r6w7n\" (UID: \"6ef6f72d-739e-4f95-83de-46f03c3dbaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n" Oct 30 18:38:26.960713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.960599 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2vkp\" (UniqueName: \"kubernetes.io/projected/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-kube-api-access-z2vkp\") pod \"maas-api-revoke-sub-79805ca9-r6w7n\" (UID: \"6ef6f72d-739e-4f95-83de-46f03c3dbaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n" Oct 30 18:38:26.960713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.960619 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-r6w7n\" (UID: \"6ef6f72d-739e-4f95-83de-46f03c3dbaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n" Oct 30 18:38:26.961209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.961189 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-r6w7n\" (UID: \"6ef6f72d-739e-4f95-83de-46f03c3dbaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n" Oct 30 18:38:26.962942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.962922 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-r6w7n\" (UID: \"6ef6f72d-739e-4f95-83de-46f03c3dbaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n" Oct 30 18:38:26.979486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:26.979463 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2vkp\" (UniqueName: \"kubernetes.io/projected/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-kube-api-access-z2vkp\") pod \"maas-api-revoke-sub-79805ca9-r6w7n\" (UID: \"6ef6f72d-739e-4f95-83de-46f03c3dbaa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n" Oct 30 18:38:27.031862 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:38:27.031820 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:38:27.157133 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:27.157060 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n" Oct 30 18:38:27.447007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:27.446977 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n"] Oct 30 18:38:27.448655 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:38:27.448624 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ef6f72d_739e_4f95_83de_46f03c3dbaa3.slice/crio-7ad95dcee685949ade9d5cdb25d89253af20898a5bbf59050b243d71c96eeac3 WatchSource:0}: Error finding container 7ad95dcee685949ade9d5cdb25d89253af20898a5bbf59050b243d71c96eeac3: Status 404 returned error can't find the container with id 7ad95dcee685949ade9d5cdb25d89253af20898a5bbf59050b243d71c96eeac3 Oct 30 18:38:27.517832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:27.517795 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n" event={"ID":"6ef6f72d-739e-4f95-83de-46f03c3dbaa3","Type":"ContainerStarted","Data":"7ad95dcee685949ade9d5cdb25d89253af20898a5bbf59050b243d71c96eeac3"} Oct 30 18:38:28.525027 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:28.524933 5232 generic.go:356] "Generic (PLEG): container finished" podID="6ef6f72d-739e-4f95-83de-46f03c3dbaa3" containerID="d2a05dfc26e6de78228c34007426a77c6ab2ecc4a6346099d9ffe27efd7b09ac" exitCode=0 Oct 30 18:38:28.525440 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:28.525024 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n" event={"ID":"6ef6f72d-739e-4f95-83de-46f03c3dbaa3","Type":"ContainerDied","Data":"d2a05dfc26e6de78228c34007426a77c6ab2ecc4a6346099d9ffe27efd7b09ac"} Oct 30 18:38:29.655904 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:29.655880 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n" Oct 30 18:38:29.683972 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:29.683945 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-maas-api-openshift-ca\") pod \"6ef6f72d-739e-4f95-83de-46f03c3dbaa3\" (UID: \"6ef6f72d-739e-4f95-83de-46f03c3dbaa3\") " Oct 30 18:38:29.684131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:29.684041 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-maas-api-xks-ca\") pod \"6ef6f72d-739e-4f95-83de-46f03c3dbaa3\" (UID: \"6ef6f72d-739e-4f95-83de-46f03c3dbaa3\") " Oct 30 18:38:29.684131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:29.684090 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-kube-api-access-z2vkp\" (UniqueName: \"kubernetes.io/projected/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-kube-api-access-z2vkp\") pod \"6ef6f72d-739e-4f95-83de-46f03c3dbaa3\" (UID: \"6ef6f72d-739e-4f95-83de-46f03c3dbaa3\") " Oct 30 18:38:29.684339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:29.684315 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-maas-api-openshift-ca" pod "6ef6f72d-739e-4f95-83de-46f03c3dbaa3" (UID: "6ef6f72d-739e-4f95-83de-46f03c3dbaa3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:29.686255 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:29.686231 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-maas-api-xks-ca" pod "6ef6f72d-739e-4f95-83de-46f03c3dbaa3" (UID: "6ef6f72d-739e-4f95-83de-46f03c3dbaa3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:29.686380 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:29.686360 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-kube-api-access-z2vkp" pod "6ef6f72d-739e-4f95-83de-46f03c3dbaa3" (UID: "6ef6f72d-739e-4f95-83de-46f03c3dbaa3"). InnerVolumeSpecName "kube-api-access-z2vkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:29.785695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:29.785621 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:38:29.785695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:29.785646 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2vkp\" (UniqueName: \"kubernetes.io/projected/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-kube-api-access-z2vkp\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:38:29.785695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:29.785656 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6ef6f72d-739e-4f95-83de-46f03c3dbaa3-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:38:30.538299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:30.538263 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n" Oct 30 18:38:30.538299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:30.538281 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n" event={"ID":"6ef6f72d-739e-4f95-83de-46f03c3dbaa3","Type":"ContainerDied","Data":"7ad95dcee685949ade9d5cdb25d89253af20898a5bbf59050b243d71c96eeac3"} Oct 30 18:38:30.538542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:30.538325 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7ad95dcee685949ade9d5cdb25d89253af20898a5bbf59050b243d71c96eeac3" Oct 30 18:38:46.864493 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:46.864456 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6"] Oct 30 18:38:46.865150 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:46.864935 5232 cpu_manager.go:414] "containerMap: removing container" podUID="6ef6f72d-739e-4f95-83de-46f03c3dbaa3" containerName="revoke-keys" Oct 30 18:38:46.865150 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:46.864954 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ef6f72d-739e-4f95-83de-46f03c3dbaa3" containerName="revoke-keys" Oct 30 18:38:46.865150 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:46.865058 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ef6f72d-739e-4f95-83de-46f03c3dbaa3" containerName="revoke-keys" Oct 30 18:38:46.868424 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:46.868402 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6" Oct 30 18:38:46.884221 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:46.884188 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:38:46.886658 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:46.886634 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6"] Oct 30 18:38:46.896096 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:46.896072 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:38:46.985117 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:46.985083 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s9fq2\" (UniqueName: \"kubernetes.io/projected/aacc7754-1a56-4c14-9e31-2afad4653145-kube-api-access-s9fq2\") pod \"maas-api-revoke-sub-0b7e8edb-rbdq6\" (UID: \"aacc7754-1a56-4c14-9e31-2afad4653145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6" Oct 30 18:38:46.985311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:46.985137 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacc7754-1a56-4c14-9e31-2afad4653145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rbdq6\" (UID: \"aacc7754-1a56-4c14-9e31-2afad4653145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6" Oct 30 18:38:46.985311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:46.985211 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacc7754-1a56-4c14-9e31-2afad4653145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rbdq6\" (UID: \"aacc7754-1a56-4c14-9e31-2afad4653145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6" Oct 30 18:38:47.086052 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:47.085999 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s9fq2\" (UniqueName: \"kubernetes.io/projected/aacc7754-1a56-4c14-9e31-2afad4653145-kube-api-access-s9fq2\") pod \"maas-api-revoke-sub-0b7e8edb-rbdq6\" (UID: \"aacc7754-1a56-4c14-9e31-2afad4653145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6" Oct 30 18:38:47.086245 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:47.086083 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacc7754-1a56-4c14-9e31-2afad4653145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rbdq6\" (UID: \"aacc7754-1a56-4c14-9e31-2afad4653145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6" Oct 30 18:38:47.086245 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:47.086123 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacc7754-1a56-4c14-9e31-2afad4653145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rbdq6\" (UID: \"aacc7754-1a56-4c14-9e31-2afad4653145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6" Oct 30 18:38:47.086786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:47.086754 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacc7754-1a56-4c14-9e31-2afad4653145-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rbdq6\" (UID: \"aacc7754-1a56-4c14-9e31-2afad4653145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6" Oct 30 18:38:47.088623 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:47.088603 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacc7754-1a56-4c14-9e31-2afad4653145-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-rbdq6\" (UID: \"aacc7754-1a56-4c14-9e31-2afad4653145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6" Oct 30 18:38:47.105480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:47.105453 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s9fq2\" (UniqueName: \"kubernetes.io/projected/aacc7754-1a56-4c14-9e31-2afad4653145-kube-api-access-s9fq2\") pod \"maas-api-revoke-sub-0b7e8edb-rbdq6\" (UID: \"aacc7754-1a56-4c14-9e31-2afad4653145\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6" Oct 30 18:38:47.181028 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:47.180963 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6" Oct 30 18:38:47.481661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:47.471101 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6"] Oct 30 18:38:47.481661 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:38:47.473467 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaacc7754_1a56_4c14_9e31_2afad4653145.slice/crio-aee446e2de8e9719b586d3a77c8d89a930673bbc287855081f810d782f8b8a64 WatchSource:0}: Error finding container aee446e2de8e9719b586d3a77c8d89a930673bbc287855081f810d782f8b8a64: Status 404 returned error can't find the container with id aee446e2de8e9719b586d3a77c8d89a930673bbc287855081f810d782f8b8a64 Oct 30 18:38:47.634723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:47.634693 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6" event={"ID":"aacc7754-1a56-4c14-9e31-2afad4653145","Type":"ContainerStarted","Data":"aee446e2de8e9719b586d3a77c8d89a930673bbc287855081f810d782f8b8a64"} Oct 30 18:38:48.640972 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:48.640934 5232 generic.go:356] "Generic (PLEG): container finished" podID="aacc7754-1a56-4c14-9e31-2afad4653145" containerID="9a093530a235dbb4c3a548ce644c90c1534787e6e54dd2b5b882d495d1c910c9" exitCode=0 Oct 30 18:38:48.641429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:48.641023 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6" event={"ID":"aacc7754-1a56-4c14-9e31-2afad4653145","Type":"ContainerDied","Data":"9a093530a235dbb4c3a548ce644c90c1534787e6e54dd2b5b882d495d1c910c9"} Oct 30 18:38:49.774313 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:49.774287 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6" Oct 30 18:38:49.914311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:49.914225 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aacc7754-1a56-4c14-9e31-2afad4653145-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacc7754-1a56-4c14-9e31-2afad4653145-maas-api-xks-ca\") pod \"aacc7754-1a56-4c14-9e31-2afad4653145\" (UID: \"aacc7754-1a56-4c14-9e31-2afad4653145\") " Oct 30 18:38:49.914311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:49.914285 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aacc7754-1a56-4c14-9e31-2afad4653145-kube-api-access-s9fq2\" (UniqueName: \"kubernetes.io/projected/aacc7754-1a56-4c14-9e31-2afad4653145-kube-api-access-s9fq2\") pod \"aacc7754-1a56-4c14-9e31-2afad4653145\" (UID: \"aacc7754-1a56-4c14-9e31-2afad4653145\") " Oct 30 18:38:49.914620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:49.914322 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aacc7754-1a56-4c14-9e31-2afad4653145-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacc7754-1a56-4c14-9e31-2afad4653145-maas-api-openshift-ca\") pod \"aacc7754-1a56-4c14-9e31-2afad4653145\" (UID: \"aacc7754-1a56-4c14-9e31-2afad4653145\") " Oct 30 18:38:49.914830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:49.914798 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aacc7754-1a56-4c14-9e31-2afad4653145-maas-api-openshift-ca" pod "aacc7754-1a56-4c14-9e31-2afad4653145" (UID: "aacc7754-1a56-4c14-9e31-2afad4653145"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:38:49.916515 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:49.916484 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aacc7754-1a56-4c14-9e31-2afad4653145-maas-api-xks-ca" pod "aacc7754-1a56-4c14-9e31-2afad4653145" (UID: "aacc7754-1a56-4c14-9e31-2afad4653145"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:38:49.916682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:49.916665 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aacc7754-1a56-4c14-9e31-2afad4653145-kube-api-access-s9fq2" pod "aacc7754-1a56-4c14-9e31-2afad4653145" (UID: "aacc7754-1a56-4c14-9e31-2afad4653145"). InnerVolumeSpecName "kube-api-access-s9fq2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:38:50.015146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:50.015109 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s9fq2\" (UniqueName: \"kubernetes.io/projected/aacc7754-1a56-4c14-9e31-2afad4653145-kube-api-access-s9fq2\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:38:50.015146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:50.015138 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aacc7754-1a56-4c14-9e31-2afad4653145-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:38:50.015146 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:50.015149 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aacc7754-1a56-4c14-9e31-2afad4653145-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:38:50.652853 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:50.652815 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6" event={"ID":"aacc7754-1a56-4c14-9e31-2afad4653145","Type":"ContainerDied","Data":"aee446e2de8e9719b586d3a77c8d89a930673bbc287855081f810d782f8b8a64"} Oct 30 18:38:50.653065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:50.652869 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aee446e2de8e9719b586d3a77c8d89a930673bbc287855081f810d782f8b8a64" Oct 30 18:38:50.653065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:50.652830 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6" Oct 30 18:38:55.478108 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:55.478079 5232 scope.go:122] "RemoveContainer" containerID="6625673d980fdedfe55766768693171e22a17794246e08d14c39d919ca7bf359" Oct 30 18:38:55.487249 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:55.487226 5232 scope.go:122] "RemoveContainer" containerID="a4b6b1e0f023dbff944bfc3801ffa9b926498cf94ec14effeff1c6a26464feab" Oct 30 18:38:55.495394 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:55.495373 5232 scope.go:122] "RemoveContainer" containerID="1c3442c0a201135f1ff6f9a7841a7369e05affdce5b82eddcafa0628566f692a" Oct 30 18:38:57.032897 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:38:57.032853 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:38:57.742985 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:57.742949 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x"] Oct 30 18:38:57.743326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:57.743313 5232 cpu_manager.go:414] "containerMap: removing container" podUID="aacc7754-1a56-4c14-9e31-2afad4653145" containerName="revoke-keys" Oct 30 18:38:57.743383 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:57.743328 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aacc7754-1a56-4c14-9e31-2afad4653145" containerName="revoke-keys" Oct 30 18:38:57.743429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:57.743392 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="aacc7754-1a56-4c14-9e31-2afad4653145" containerName="revoke-keys" Oct 30 18:38:57.747781 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:57.747762 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x" Oct 30 18:38:57.766007 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:57.765974 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x"] Oct 30 18:38:57.775338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:57.775307 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:38:57.775338 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:57.775335 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:38:57.884791 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:57.884753 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ndks7\" (UniqueName: \"kubernetes.io/projected/3bb8382e-788d-4ffb-8791-efb61a8296b9-kube-api-access-ndks7\") pod \"maas-api-revoke-sub-3d4156f1-8467x\" (UID: \"3bb8382e-788d-4ffb-8791-efb61a8296b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x" Oct 30 18:38:57.885005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:57.884824 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bb8382e-788d-4ffb-8791-efb61a8296b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8467x\" (UID: \"3bb8382e-788d-4ffb-8791-efb61a8296b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x" Oct 30 18:38:57.885005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:57.884952 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bb8382e-788d-4ffb-8791-efb61a8296b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8467x\" (UID: \"3bb8382e-788d-4ffb-8791-efb61a8296b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x" Oct 30 18:38:57.986082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:57.986045 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ndks7\" (UniqueName: \"kubernetes.io/projected/3bb8382e-788d-4ffb-8791-efb61a8296b9-kube-api-access-ndks7\") pod \"maas-api-revoke-sub-3d4156f1-8467x\" (UID: \"3bb8382e-788d-4ffb-8791-efb61a8296b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x" Oct 30 18:38:57.986294 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:57.986119 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bb8382e-788d-4ffb-8791-efb61a8296b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8467x\" (UID: \"3bb8382e-788d-4ffb-8791-efb61a8296b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x" Oct 30 18:38:57.986294 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:57.986156 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bb8382e-788d-4ffb-8791-efb61a8296b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8467x\" (UID: \"3bb8382e-788d-4ffb-8791-efb61a8296b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x" Oct 30 18:38:57.986951 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:57.986929 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bb8382e-788d-4ffb-8791-efb61a8296b9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8467x\" (UID: \"3bb8382e-788d-4ffb-8791-efb61a8296b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x" Oct 30 18:38:57.988733 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:57.988707 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bb8382e-788d-4ffb-8791-efb61a8296b9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8467x\" (UID: \"3bb8382e-788d-4ffb-8791-efb61a8296b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x" Oct 30 18:38:58.005432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:58.005364 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ndks7\" (UniqueName: \"kubernetes.io/projected/3bb8382e-788d-4ffb-8791-efb61a8296b9-kube-api-access-ndks7\") pod \"maas-api-revoke-sub-3d4156f1-8467x\" (UID: \"3bb8382e-788d-4ffb-8791-efb61a8296b9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x" Oct 30 18:38:58.060554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:58.060497 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x" Oct 30 18:38:58.349801 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:58.349777 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x"] Oct 30 18:38:58.352094 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:38:58.352053 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3bb8382e_788d_4ffb_8791_efb61a8296b9.slice/crio-cb6c90093fa8a7e791d8b6b3a2faffc216042b422720fc7789a5d1ccbf7ffab5 WatchSource:0}: Error finding container cb6c90093fa8a7e791d8b6b3a2faffc216042b422720fc7789a5d1ccbf7ffab5: Status 404 returned error can't find the container with id cb6c90093fa8a7e791d8b6b3a2faffc216042b422720fc7789a5d1ccbf7ffab5 Oct 30 18:38:58.694383 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:58.694348 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x" event={"ID":"3bb8382e-788d-4ffb-8791-efb61a8296b9","Type":"ContainerStarted","Data":"be1ec851f185699f10cb580af0180e9c0db238c7b187bcabfe417d00ad9a0867"} Oct 30 18:38:58.694583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:58.694398 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x" event={"ID":"3bb8382e-788d-4ffb-8791-efb61a8296b9","Type":"ContainerStarted","Data":"cb6c90093fa8a7e791d8b6b3a2faffc216042b422720fc7789a5d1ccbf7ffab5"} Oct 30 18:38:58.732902 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:58.732854 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x" podStartSLOduration=1.732840108 podStartE2EDuration="1.732840108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:38:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:38:58.73144934 +0000 UTC m=+2238.047869662" watchObservedRunningTime="2026-10-30 18:38:58.732840108 +0000 UTC m=+2238.049260428" Oct 30 18:38:59.701106 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:59.701067 5232 generic.go:356] "Generic (PLEG): container finished" podID="3bb8382e-788d-4ffb-8791-efb61a8296b9" containerID="be1ec851f185699f10cb580af0180e9c0db238c7b187bcabfe417d00ad9a0867" exitCode=0 Oct 30 18:38:59.701534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:38:59.701148 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x" event={"ID":"3bb8382e-788d-4ffb-8791-efb61a8296b9","Type":"ContainerDied","Data":"be1ec851f185699f10cb580af0180e9c0db238c7b187bcabfe417d00ad9a0867"} Oct 30 18:39:00.836620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:00.836596 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x" Oct 30 18:39:01.016136 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:01.016089 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3bb8382e-788d-4ffb-8791-efb61a8296b9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bb8382e-788d-4ffb-8791-efb61a8296b9-maas-api-openshift-ca\") pod \"3bb8382e-788d-4ffb-8791-efb61a8296b9\" (UID: \"3bb8382e-788d-4ffb-8791-efb61a8296b9\") " Oct 30 18:39:01.016136 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:01.016147 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3bb8382e-788d-4ffb-8791-efb61a8296b9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bb8382e-788d-4ffb-8791-efb61a8296b9-maas-api-xks-ca\") pod \"3bb8382e-788d-4ffb-8791-efb61a8296b9\" (UID: \"3bb8382e-788d-4ffb-8791-efb61a8296b9\") " Oct 30 18:39:01.016378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:01.016268 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3bb8382e-788d-4ffb-8791-efb61a8296b9-kube-api-access-ndks7\" (UniqueName: \"kubernetes.io/projected/3bb8382e-788d-4ffb-8791-efb61a8296b9-kube-api-access-ndks7\") pod \"3bb8382e-788d-4ffb-8791-efb61a8296b9\" (UID: \"3bb8382e-788d-4ffb-8791-efb61a8296b9\") " Oct 30 18:39:01.016480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:01.016457 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3bb8382e-788d-4ffb-8791-efb61a8296b9-maas-api-openshift-ca" pod "3bb8382e-788d-4ffb-8791-efb61a8296b9" (UID: "3bb8382e-788d-4ffb-8791-efb61a8296b9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:01.018544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:01.018496 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3bb8382e-788d-4ffb-8791-efb61a8296b9-maas-api-xks-ca" pod "3bb8382e-788d-4ffb-8791-efb61a8296b9" (UID: "3bb8382e-788d-4ffb-8791-efb61a8296b9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:01.018544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:01.018528 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3bb8382e-788d-4ffb-8791-efb61a8296b9-kube-api-access-ndks7" pod "3bb8382e-788d-4ffb-8791-efb61a8296b9" (UID: "3bb8382e-788d-4ffb-8791-efb61a8296b9"). InnerVolumeSpecName "kube-api-access-ndks7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:01.117988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:01.117938 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ndks7\" (UniqueName: \"kubernetes.io/projected/3bb8382e-788d-4ffb-8791-efb61a8296b9-kube-api-access-ndks7\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:39:01.117988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:01.117983 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3bb8382e-788d-4ffb-8791-efb61a8296b9-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:39:01.117988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:01.117996 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3bb8382e-788d-4ffb-8791-efb61a8296b9-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:39:01.712135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:01.712104 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x" Oct 30 18:39:01.712318 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:01.712140 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x" event={"ID":"3bb8382e-788d-4ffb-8791-efb61a8296b9","Type":"ContainerDied","Data":"cb6c90093fa8a7e791d8b6b3a2faffc216042b422720fc7789a5d1ccbf7ffab5"} Oct 30 18:39:01.712318 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:01.712217 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cb6c90093fa8a7e791d8b6b3a2faffc216042b422720fc7789a5d1ccbf7ffab5" Oct 30 18:39:02.025457 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:02.025422 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt"] Oct 30 18:39:02.028372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:02.028348 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a93233a0-kjrwt"] Oct 30 18:39:02.800471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:02.800434 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf142398-5923-4658-bdf2-cd3c76b0ef93" path="/var/lib/kubelet/pods/cf142398-5923-4658-bdf2-cd3c76b0ef93/volumes" Oct 30 18:39:22.162210 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.162118 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x"] Oct 30 18:39:22.174154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.174124 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8467x"] Oct 30 18:39:22.206501 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.206462 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq"] Oct 30 18:39:22.206909 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.206892 5232 cpu_manager.go:414] "containerMap: removing container" podUID="3bb8382e-788d-4ffb-8791-efb61a8296b9" containerName="revoke-keys" Oct 30 18:39:22.206909 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.206910 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3bb8382e-788d-4ffb-8791-efb61a8296b9" containerName="revoke-keys" Oct 30 18:39:22.207013 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.206989 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="3bb8382e-788d-4ffb-8791-efb61a8296b9" containerName="revoke-keys" Oct 30 18:39:22.210148 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.210128 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq" Oct 30 18:39:22.225486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.225452 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:39:22.225647 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.225584 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:39:22.228279 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.228253 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq"] Oct 30 18:39:22.255729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.255696 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gzhrq\" (UID: \"b38e70ff-0b0c-4acd-8d84-6d698b19a9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq" Oct 30 18:39:22.255892 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.255742 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gzhrq\" (UID: \"b38e70ff-0b0c-4acd-8d84-6d698b19a9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq" Oct 30 18:39:22.255892 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.255824 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nwhc2\" (UniqueName: \"kubernetes.io/projected/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-kube-api-access-nwhc2\") pod \"maas-api-revoke-sub-3d4156f1-gzhrq\" (UID: \"b38e70ff-0b0c-4acd-8d84-6d698b19a9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq" Oct 30 18:39:22.357078 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.357043 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gzhrq\" (UID: \"b38e70ff-0b0c-4acd-8d84-6d698b19a9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq" Oct 30 18:39:22.357289 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.357091 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nwhc2\" (UniqueName: \"kubernetes.io/projected/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-kube-api-access-nwhc2\") pod \"maas-api-revoke-sub-3d4156f1-gzhrq\" (UID: \"b38e70ff-0b0c-4acd-8d84-6d698b19a9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq" Oct 30 18:39:22.357289 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.357169 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gzhrq\" (UID: \"b38e70ff-0b0c-4acd-8d84-6d698b19a9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq" Oct 30 18:39:22.357749 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.357721 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-gzhrq\" (UID: \"b38e70ff-0b0c-4acd-8d84-6d698b19a9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq" Oct 30 18:39:22.359655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.359637 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-gzhrq\" (UID: \"b38e70ff-0b0c-4acd-8d84-6d698b19a9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq" Oct 30 18:39:22.377761 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.377730 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nwhc2\" (UniqueName: \"kubernetes.io/projected/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-kube-api-access-nwhc2\") pod \"maas-api-revoke-sub-3d4156f1-gzhrq\" (UID: \"b38e70ff-0b0c-4acd-8d84-6d698b19a9c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq" Oct 30 18:39:22.522683 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.522644 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq" Oct 30 18:39:22.800336 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.800261 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3bb8382e-788d-4ffb-8791-efb61a8296b9" path="/var/lib/kubelet/pods/3bb8382e-788d-4ffb-8791-efb61a8296b9/volumes" Oct 30 18:39:22.816362 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.816335 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq"] Oct 30 18:39:22.818524 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:39:22.818483 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb38e70ff_0b0c_4acd_8d84_6d698b19a9c4.slice/crio-9359c49d8547eb176f4058afd3557da5301b1701b1ffced854727b01e8c799d3 WatchSource:0}: Error finding container 9359c49d8547eb176f4058afd3557da5301b1701b1ffced854727b01e8c799d3: Status 404 returned error can't find the container with id 9359c49d8547eb176f4058afd3557da5301b1701b1ffced854727b01e8c799d3 Oct 30 18:39:22.884098 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:22.884064 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq" event={"ID":"b38e70ff-0b0c-4acd-8d84-6d698b19a9c4","Type":"ContainerStarted","Data":"9359c49d8547eb176f4058afd3557da5301b1701b1ffced854727b01e8c799d3"} Oct 30 18:39:23.890415 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:23.890323 5232 generic.go:356] "Generic (PLEG): container finished" podID="b38e70ff-0b0c-4acd-8d84-6d698b19a9c4" containerID="397ec5bbcb82f4f0ad5a46ece52120f6cb21ef1898ccc4ed3990809c4c5d0f88" exitCode=0 Oct 30 18:39:23.890831 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:23.890412 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq" event={"ID":"b38e70ff-0b0c-4acd-8d84-6d698b19a9c4","Type":"ContainerDied","Data":"397ec5bbcb82f4f0ad5a46ece52120f6cb21ef1898ccc4ed3990809c4c5d0f88"} Oct 30 18:39:25.033595 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:25.033568 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq" Oct 30 18:39:25.079746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:25.079716 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-maas-api-xks-ca\") pod \"b38e70ff-0b0c-4acd-8d84-6d698b19a9c4\" (UID: \"b38e70ff-0b0c-4acd-8d84-6d698b19a9c4\") " Oct 30 18:39:25.079935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:25.079777 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-kube-api-access-nwhc2\" (UniqueName: \"kubernetes.io/projected/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-kube-api-access-nwhc2\") pod \"b38e70ff-0b0c-4acd-8d84-6d698b19a9c4\" (UID: \"b38e70ff-0b0c-4acd-8d84-6d698b19a9c4\") " Oct 30 18:39:25.079935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:25.079835 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-maas-api-openshift-ca\") pod \"b38e70ff-0b0c-4acd-8d84-6d698b19a9c4\" (UID: \"b38e70ff-0b0c-4acd-8d84-6d698b19a9c4\") " Oct 30 18:39:25.080248 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:25.080225 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-maas-api-openshift-ca" pod "b38e70ff-0b0c-4acd-8d84-6d698b19a9c4" (UID: "b38e70ff-0b0c-4acd-8d84-6d698b19a9c4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:25.082050 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:25.082026 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-kube-api-access-nwhc2" pod "b38e70ff-0b0c-4acd-8d84-6d698b19a9c4" (UID: "b38e70ff-0b0c-4acd-8d84-6d698b19a9c4"). InnerVolumeSpecName "kube-api-access-nwhc2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:25.082195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:25.082176 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-maas-api-xks-ca" pod "b38e70ff-0b0c-4acd-8d84-6d698b19a9c4" (UID: "b38e70ff-0b0c-4acd-8d84-6d698b19a9c4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:25.181070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:25.180980 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:39:25.181070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:25.181014 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:39:25.181070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:25.181025 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nwhc2\" (UniqueName: \"kubernetes.io/projected/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4-kube-api-access-nwhc2\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:39:25.910525 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:25.910491 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq" Oct 30 18:39:25.910698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:25.910536 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq" event={"ID":"b38e70ff-0b0c-4acd-8d84-6d698b19a9c4","Type":"ContainerDied","Data":"9359c49d8547eb176f4058afd3557da5301b1701b1ffced854727b01e8c799d3"} Oct 30 18:39:25.910698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:25.910591 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9359c49d8547eb176f4058afd3557da5301b1701b1ffced854727b01e8c799d3" Oct 30 18:39:27.032636 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:39:27.032600 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:39:45.825534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:45.824610 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-zzlvr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:39:45.825534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:45.824693 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-zzlvr" podUID="b075648b-215b-4d0e-b3dc-41ea5ed46ac2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.246:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:39:46.039535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:46.036740 5232 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-77f556d647-kbsdt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:39:46.039535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:46.036801 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-77f556d647-kbsdt" podUID="92403b9c-a622-457e-8b9b-a92f86f93b62" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.0.247:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:39:50.170290 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.170256 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp"] Oct 30 18:39:50.170684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.170664 5232 cpu_manager.go:414] "containerMap: removing container" podUID="b38e70ff-0b0c-4acd-8d84-6d698b19a9c4" containerName="revoke-keys" Oct 30 18:39:50.170684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.170680 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b38e70ff-0b0c-4acd-8d84-6d698b19a9c4" containerName="revoke-keys" Oct 30 18:39:50.170768 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.170746 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="b38e70ff-0b0c-4acd-8d84-6d698b19a9c4" containerName="revoke-keys" Oct 30 18:39:50.173908 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.173877 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp" Oct 30 18:39:50.201587 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.201556 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:39:50.201587 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.201564 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:39:50.204806 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.204766 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp"] Oct 30 18:39:50.246878 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.246838 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m5xsn\" (UniqueName: \"kubernetes.io/projected/992364e7-f08d-4aac-a893-8c91f303948f-kube-api-access-m5xsn\") pod \"maas-api-revoke-sub-e2fc9c5b-6frhp\" (UID: \"992364e7-f08d-4aac-a893-8c91f303948f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp" Oct 30 18:39:50.247082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.246985 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/992364e7-f08d-4aac-a893-8c91f303948f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6frhp\" (UID: \"992364e7-f08d-4aac-a893-8c91f303948f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp" Oct 30 18:39:50.247158 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.247102 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/992364e7-f08d-4aac-a893-8c91f303948f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6frhp\" (UID: \"992364e7-f08d-4aac-a893-8c91f303948f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp" Oct 30 18:39:50.348374 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.348322 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m5xsn\" (UniqueName: \"kubernetes.io/projected/992364e7-f08d-4aac-a893-8c91f303948f-kube-api-access-m5xsn\") pod \"maas-api-revoke-sub-e2fc9c5b-6frhp\" (UID: \"992364e7-f08d-4aac-a893-8c91f303948f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp" Oct 30 18:39:50.348635 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.348410 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/992364e7-f08d-4aac-a893-8c91f303948f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6frhp\" (UID: \"992364e7-f08d-4aac-a893-8c91f303948f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp" Oct 30 18:39:50.348635 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.348468 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/992364e7-f08d-4aac-a893-8c91f303948f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6frhp\" (UID: \"992364e7-f08d-4aac-a893-8c91f303948f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp" Oct 30 18:39:50.349156 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.349134 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/992364e7-f08d-4aac-a893-8c91f303948f-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6frhp\" (UID: \"992364e7-f08d-4aac-a893-8c91f303948f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp" Oct 30 18:39:50.350924 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.350902 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/992364e7-f08d-4aac-a893-8c91f303948f-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-6frhp\" (UID: \"992364e7-f08d-4aac-a893-8c91f303948f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp" Oct 30 18:39:50.368443 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.368409 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m5xsn\" (UniqueName: \"kubernetes.io/projected/992364e7-f08d-4aac-a893-8c91f303948f-kube-api-access-m5xsn\") pod \"maas-api-revoke-sub-e2fc9c5b-6frhp\" (UID: \"992364e7-f08d-4aac-a893-8c91f303948f\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp" Oct 30 18:39:50.487564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.487530 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp" Oct 30 18:39:50.778201 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:50.778171 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp"] Oct 30 18:39:50.780062 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:39:50.780035 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod992364e7_f08d_4aac_a893_8c91f303948f.slice/crio-e6a64779b175d6be57c140cd44972e8cfc70914e782feedea4a69735f2de5054 WatchSource:0}: Error finding container e6a64779b175d6be57c140cd44972e8cfc70914e782feedea4a69735f2de5054: Status 404 returned error can't find the container with id e6a64779b175d6be57c140cd44972e8cfc70914e782feedea4a69735f2de5054 Oct 30 18:39:51.065986 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:51.065902 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp" event={"ID":"992364e7-f08d-4aac-a893-8c91f303948f","Type":"ContainerStarted","Data":"e6a64779b175d6be57c140cd44972e8cfc70914e782feedea4a69735f2de5054"} Oct 30 18:39:52.072229 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:52.072193 5232 generic.go:356] "Generic (PLEG): container finished" podID="992364e7-f08d-4aac-a893-8c91f303948f" containerID="d6c753ae7c93cec5309f8ef36d442b519f219808091ef815bd3f27b23d7b044a" exitCode=0 Oct 30 18:39:52.072641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:52.072277 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp" event={"ID":"992364e7-f08d-4aac-a893-8c91f303948f","Type":"ContainerDied","Data":"d6c753ae7c93cec5309f8ef36d442b519f219808091ef815bd3f27b23d7b044a"} Oct 30 18:39:53.025046 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:53.025010 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6"] Oct 30 18:39:53.037152 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:53.037124 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-85237e45-24pt6"] Oct 30 18:39:53.209464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:53.209439 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp" Oct 30 18:39:53.276832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:53.276752 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/992364e7-f08d-4aac-a893-8c91f303948f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/992364e7-f08d-4aac-a893-8c91f303948f-maas-api-openshift-ca\") pod \"992364e7-f08d-4aac-a893-8c91f303948f\" (UID: \"992364e7-f08d-4aac-a893-8c91f303948f\") " Oct 30 18:39:53.276832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:53.276806 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/992364e7-f08d-4aac-a893-8c91f303948f-kube-api-access-m5xsn\" (UniqueName: \"kubernetes.io/projected/992364e7-f08d-4aac-a893-8c91f303948f-kube-api-access-m5xsn\") pod \"992364e7-f08d-4aac-a893-8c91f303948f\" (UID: \"992364e7-f08d-4aac-a893-8c91f303948f\") " Oct 30 18:39:53.277084 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:53.276848 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/992364e7-f08d-4aac-a893-8c91f303948f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/992364e7-f08d-4aac-a893-8c91f303948f-maas-api-xks-ca\") pod \"992364e7-f08d-4aac-a893-8c91f303948f\" (UID: \"992364e7-f08d-4aac-a893-8c91f303948f\") " Oct 30 18:39:53.277182 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:53.277152 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/992364e7-f08d-4aac-a893-8c91f303948f-maas-api-openshift-ca" pod "992364e7-f08d-4aac-a893-8c91f303948f" (UID: "992364e7-f08d-4aac-a893-8c91f303948f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:39:53.279029 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:53.279006 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/992364e7-f08d-4aac-a893-8c91f303948f-kube-api-access-m5xsn" pod "992364e7-f08d-4aac-a893-8c91f303948f" (UID: "992364e7-f08d-4aac-a893-8c91f303948f"). InnerVolumeSpecName "kube-api-access-m5xsn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:39:53.279143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:53.279057 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/992364e7-f08d-4aac-a893-8c91f303948f-maas-api-xks-ca" pod "992364e7-f08d-4aac-a893-8c91f303948f" (UID: "992364e7-f08d-4aac-a893-8c91f303948f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:39:53.377477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:53.377438 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/992364e7-f08d-4aac-a893-8c91f303948f-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:39:53.377477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:53.377469 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m5xsn\" (UniqueName: \"kubernetes.io/projected/992364e7-f08d-4aac-a893-8c91f303948f-kube-api-access-m5xsn\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:39:53.377477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:53.377480 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/992364e7-f08d-4aac-a893-8c91f303948f-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:39:54.084610 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:54.084577 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp" Oct 30 18:39:54.084610 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:54.084584 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp" event={"ID":"992364e7-f08d-4aac-a893-8c91f303948f","Type":"ContainerDied","Data":"e6a64779b175d6be57c140cd44972e8cfc70914e782feedea4a69735f2de5054"} Oct 30 18:39:54.084873 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:54.084637 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e6a64779b175d6be57c140cd44972e8cfc70914e782feedea4a69735f2de5054" Oct 30 18:39:54.800867 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:54.800834 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59aec578-9be5-4f28-ac32-26270ee0f22e" path="/var/lib/kubelet/pods/59aec578-9be5-4f28-ac32-26270ee0f22e/volumes" Oct 30 18:39:55.522826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:55.522794 5232 scope.go:122] "RemoveContainer" containerID="64eea3b4936d5047087fac8a2b3af95e815ca831afd7136c1e894d077a2bb943" Oct 30 18:39:55.532130 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:55.532106 5232 scope.go:122] "RemoveContainer" containerID="993e8e5bfb6e39aeeba2291eae428d565579c3c63d15b154d9b8dc1911533d31" Oct 30 18:39:55.542992 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:55.542971 5232 scope.go:122] "RemoveContainer" containerID="e5d7f1164af3e13b2ecf4a2d874fefcc30e83e28faedd7e7916450d519945fbc" Oct 30 18:39:55.551631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:55.551607 5232 scope.go:122] "RemoveContainer" containerID="a50a7c20e45f9c7c1d559a1e690668c2549b34dfce1f77215973b239c78e4ffd" Oct 30 18:39:55.559890 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:55.559864 5232 scope.go:122] "RemoveContainer" containerID="df666ac84f97e95aa8ecd863c473cc46d29bf1e8c0dc342467c24ccb8fa1e151" Oct 30 18:39:55.568302 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:39:55.568274 5232 scope.go:122] "RemoveContainer" containerID="a17ecb82cc82b1edc1205260b8d297a7128c91e6b890218e25cac40ceb15d023" Oct 30 18:39:57.033419 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:39:57.033382 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:40:00.498483 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:00.498443 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct"] Oct 30 18:40:00.498918 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:00.498820 5232 cpu_manager.go:414] "containerMap: removing container" podUID="992364e7-f08d-4aac-a893-8c91f303948f" containerName="revoke-keys" Oct 30 18:40:00.498918 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:00.498835 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="992364e7-f08d-4aac-a893-8c91f303948f" containerName="revoke-keys" Oct 30 18:40:00.499016 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:00.498921 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="992364e7-f08d-4aac-a893-8c91f303948f" containerName="revoke-keys" Oct 30 18:40:00.504048 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:00.504025 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct" Oct 30 18:40:00.519626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:00.519570 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct"] Oct 30 18:40:00.529525 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:00.529478 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:40:00.529694 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:00.529492 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:40:00.644096 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:00.644059 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-9cqct\" (UID: \"d55fc27b-929d-4a6f-8930-ca3008ae0cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct" Oct 30 18:40:00.644300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:00.644136 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6cb7\" (UniqueName: \"kubernetes.io/projected/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-kube-api-access-w6cb7\") pod \"maas-api-revoke-sub-5a29efb3-9cqct\" (UID: \"d55fc27b-929d-4a6f-8930-ca3008ae0cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct" Oct 30 18:40:00.644300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:00.644215 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-9cqct\" (UID: \"d55fc27b-929d-4a6f-8930-ca3008ae0cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct" Oct 30 18:40:00.745004 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:00.744960 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6cb7\" (UniqueName: \"kubernetes.io/projected/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-kube-api-access-w6cb7\") pod \"maas-api-revoke-sub-5a29efb3-9cqct\" (UID: \"d55fc27b-929d-4a6f-8930-ca3008ae0cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct" Oct 30 18:40:00.745004 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:00.745012 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-9cqct\" (UID: \"d55fc27b-929d-4a6f-8930-ca3008ae0cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct" Oct 30 18:40:00.745292 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:00.745090 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-9cqct\" (UID: \"d55fc27b-929d-4a6f-8930-ca3008ae0cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct" Oct 30 18:40:00.745737 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:00.745713 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-9cqct\" (UID: \"d55fc27b-929d-4a6f-8930-ca3008ae0cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct" Oct 30 18:40:00.747640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:00.747618 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-9cqct\" (UID: \"d55fc27b-929d-4a6f-8930-ca3008ae0cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct" Oct 30 18:40:00.763899 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:00.763843 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6cb7\" (UniqueName: \"kubernetes.io/projected/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-kube-api-access-w6cb7\") pod \"maas-api-revoke-sub-5a29efb3-9cqct\" (UID: \"d55fc27b-929d-4a6f-8930-ca3008ae0cd0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct" Oct 30 18:40:00.816864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:00.816824 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct" Oct 30 18:40:01.104949 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:01.104924 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct"] Oct 30 18:40:01.106833 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:40:01.106804 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd55fc27b_929d_4a6f_8930_ca3008ae0cd0.slice/crio-1c05addb00deb0e8fbf98965e41c18f7690f505f03d7d13eb16f5f696e17f4ed WatchSource:0}: Error finding container 1c05addb00deb0e8fbf98965e41c18f7690f505f03d7d13eb16f5f696e17f4ed: Status 404 returned error can't find the container with id 1c05addb00deb0e8fbf98965e41c18f7690f505f03d7d13eb16f5f696e17f4ed Oct 30 18:40:01.127758 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:01.127729 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct" event={"ID":"d55fc27b-929d-4a6f-8930-ca3008ae0cd0","Type":"ContainerStarted","Data":"1c05addb00deb0e8fbf98965e41c18f7690f505f03d7d13eb16f5f696e17f4ed"} Oct 30 18:40:02.133833 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:02.133795 5232 generic.go:356] "Generic (PLEG): container finished" podID="d55fc27b-929d-4a6f-8930-ca3008ae0cd0" containerID="8975f10c895a5c2ed1aa80c0ae4a4faaedfe935fe80ec94cab0e51f72670f4e2" exitCode=0 Oct 30 18:40:02.134217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:02.133891 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct" event={"ID":"d55fc27b-929d-4a6f-8930-ca3008ae0cd0","Type":"ContainerDied","Data":"8975f10c895a5c2ed1aa80c0ae4a4faaedfe935fe80ec94cab0e51f72670f4e2"} Oct 30 18:40:03.507541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:03.505869 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct" Oct 30 18:40:03.741592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:03.734019 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-kube-api-access-w6cb7\" (UniqueName: \"kubernetes.io/projected/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-kube-api-access-w6cb7\") pod \"d55fc27b-929d-4a6f-8930-ca3008ae0cd0\" (UID: \"d55fc27b-929d-4a6f-8930-ca3008ae0cd0\") " Oct 30 18:40:03.792935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:03.792901 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-maas-api-xks-ca\") pod \"d55fc27b-929d-4a6f-8930-ca3008ae0cd0\" (UID: \"d55fc27b-929d-4a6f-8930-ca3008ae0cd0\") " Oct 30 18:40:03.793230 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:03.793209 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-maas-api-openshift-ca\") pod \"d55fc27b-929d-4a6f-8930-ca3008ae0cd0\" (UID: \"d55fc27b-929d-4a6f-8930-ca3008ae0cd0\") " Oct 30 18:40:03.795328 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:03.795301 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-maas-api-openshift-ca" pod "d55fc27b-929d-4a6f-8930-ca3008ae0cd0" (UID: "d55fc27b-929d-4a6f-8930-ca3008ae0cd0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:03.797127 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:03.797083 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-kube-api-access-w6cb7" pod "d55fc27b-929d-4a6f-8930-ca3008ae0cd0" (UID: "d55fc27b-929d-4a6f-8930-ca3008ae0cd0"). InnerVolumeSpecName "kube-api-access-w6cb7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:03.806725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:03.806675 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-maas-api-xks-ca" pod "d55fc27b-929d-4a6f-8930-ca3008ae0cd0" (UID: "d55fc27b-929d-4a6f-8930-ca3008ae0cd0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:03.896125 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:03.896084 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:40:03.896361 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:03.896342 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:40:03.896466 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:03.896452 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6cb7\" (UniqueName: \"kubernetes.io/projected/d55fc27b-929d-4a6f-8930-ca3008ae0cd0-kube-api-access-w6cb7\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:40:04.199883 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:04.199777 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct" event={"ID":"d55fc27b-929d-4a6f-8930-ca3008ae0cd0","Type":"ContainerDied","Data":"1c05addb00deb0e8fbf98965e41c18f7690f505f03d7d13eb16f5f696e17f4ed"} Oct 30 18:40:04.200166 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:04.200135 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1c05addb00deb0e8fbf98965e41c18f7690f505f03d7d13eb16f5f696e17f4ed" Oct 30 18:40:04.200387 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:04.200371 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct" Oct 30 18:40:19.558982 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.558940 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq"] Oct 30 18:40:19.571038 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.571008 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-gzhrq"] Oct 30 18:40:19.599492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.599457 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4"] Oct 30 18:40:19.599826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.599814 5232 cpu_manager.go:414] "containerMap: removing container" podUID="d55fc27b-929d-4a6f-8930-ca3008ae0cd0" containerName="revoke-keys" Oct 30 18:40:19.599889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.599828 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d55fc27b-929d-4a6f-8930-ca3008ae0cd0" containerName="revoke-keys" Oct 30 18:40:19.599924 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.599907 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="d55fc27b-929d-4a6f-8930-ca3008ae0cd0" containerName="revoke-keys" Oct 30 18:40:19.602735 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.602714 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4" Oct 30 18:40:19.621708 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.621656 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:40:19.621893 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.621722 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:40:19.622339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.622312 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4"] Oct 30 18:40:19.682344 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.682306 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61978b73-f8cf-4e4a-8314-4885a29166ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b9wk4\" (UID: \"61978b73-f8cf-4e4a-8314-4885a29166ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4" Oct 30 18:40:19.682567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.682441 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61978b73-f8cf-4e4a-8314-4885a29166ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b9wk4\" (UID: \"61978b73-f8cf-4e4a-8314-4885a29166ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4" Oct 30 18:40:19.682567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.682476 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkhmp\" (UniqueName: \"kubernetes.io/projected/61978b73-f8cf-4e4a-8314-4885a29166ff-kube-api-access-rkhmp\") pod \"maas-api-revoke-sub-3d4156f1-b9wk4\" (UID: \"61978b73-f8cf-4e4a-8314-4885a29166ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4" Oct 30 18:40:19.783711 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.783669 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61978b73-f8cf-4e4a-8314-4885a29166ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b9wk4\" (UID: \"61978b73-f8cf-4e4a-8314-4885a29166ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4" Oct 30 18:40:19.783901 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.783754 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61978b73-f8cf-4e4a-8314-4885a29166ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b9wk4\" (UID: \"61978b73-f8cf-4e4a-8314-4885a29166ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4" Oct 30 18:40:19.783901 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.783777 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkhmp\" (UniqueName: \"kubernetes.io/projected/61978b73-f8cf-4e4a-8314-4885a29166ff-kube-api-access-rkhmp\") pod \"maas-api-revoke-sub-3d4156f1-b9wk4\" (UID: \"61978b73-f8cf-4e4a-8314-4885a29166ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4" Oct 30 18:40:19.784434 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.784410 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61978b73-f8cf-4e4a-8314-4885a29166ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-b9wk4\" (UID: \"61978b73-f8cf-4e4a-8314-4885a29166ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4" Oct 30 18:40:19.786194 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.786174 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61978b73-f8cf-4e4a-8314-4885a29166ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-b9wk4\" (UID: \"61978b73-f8cf-4e4a-8314-4885a29166ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4" Oct 30 18:40:19.802802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.802776 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkhmp\" (UniqueName: \"kubernetes.io/projected/61978b73-f8cf-4e4a-8314-4885a29166ff-kube-api-access-rkhmp\") pod \"maas-api-revoke-sub-3d4156f1-b9wk4\" (UID: \"61978b73-f8cf-4e4a-8314-4885a29166ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4" Oct 30 18:40:19.915706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:19.915630 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4" Oct 30 18:40:20.208088 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:20.208063 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4"] Oct 30 18:40:20.209919 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:40:20.209890 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61978b73_f8cf_4e4a_8314_4885a29166ff.slice/crio-5b6222f81e4bc29facb699d31400d4dd0172c36bb3589041a787b552abca35d4 WatchSource:0}: Error finding container 5b6222f81e4bc29facb699d31400d4dd0172c36bb3589041a787b552abca35d4: Status 404 returned error can't find the container with id 5b6222f81e4bc29facb699d31400d4dd0172c36bb3589041a787b552abca35d4 Oct 30 18:40:20.277389 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:20.277354 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4" event={"ID":"61978b73-f8cf-4e4a-8314-4885a29166ff","Type":"ContainerStarted","Data":"5b6222f81e4bc29facb699d31400d4dd0172c36bb3589041a787b552abca35d4"} Oct 30 18:40:20.800658 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:20.800582 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b38e70ff-0b0c-4acd-8d84-6d698b19a9c4" path="/var/lib/kubelet/pods/b38e70ff-0b0c-4acd-8d84-6d698b19a9c4/volumes" Oct 30 18:40:21.284781 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:21.284746 5232 generic.go:356] "Generic (PLEG): container finished" podID="61978b73-f8cf-4e4a-8314-4885a29166ff" containerID="7831fbafcdc71c8e928b4eeea027283ab14dad5b6995d5f0d4f9aa0f3ddd2f96" exitCode=0 Oct 30 18:40:21.284976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:21.284837 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4" event={"ID":"61978b73-f8cf-4e4a-8314-4885a29166ff","Type":"ContainerDied","Data":"7831fbafcdc71c8e928b4eeea027283ab14dad5b6995d5f0d4f9aa0f3ddd2f96"} Oct 30 18:40:22.421640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:22.421616 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4" Oct 30 18:40:22.511419 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:22.511386 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61978b73-f8cf-4e4a-8314-4885a29166ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61978b73-f8cf-4e4a-8314-4885a29166ff-maas-api-openshift-ca\") pod \"61978b73-f8cf-4e4a-8314-4885a29166ff\" (UID: \"61978b73-f8cf-4e4a-8314-4885a29166ff\") " Oct 30 18:40:22.511676 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:22.511429 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/61978b73-f8cf-4e4a-8314-4885a29166ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61978b73-f8cf-4e4a-8314-4885a29166ff-maas-api-xks-ca\") pod \"61978b73-f8cf-4e4a-8314-4885a29166ff\" (UID: \"61978b73-f8cf-4e4a-8314-4885a29166ff\") " Oct 30 18:40:22.511676 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:22.511543 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61978b73-f8cf-4e4a-8314-4885a29166ff-kube-api-access-rkhmp\" (UniqueName: \"kubernetes.io/projected/61978b73-f8cf-4e4a-8314-4885a29166ff-kube-api-access-rkhmp\") pod \"61978b73-f8cf-4e4a-8314-4885a29166ff\" (UID: \"61978b73-f8cf-4e4a-8314-4885a29166ff\") " Oct 30 18:40:22.511867 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:22.511844 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61978b73-f8cf-4e4a-8314-4885a29166ff-maas-api-openshift-ca" pod "61978b73-f8cf-4e4a-8314-4885a29166ff" (UID: "61978b73-f8cf-4e4a-8314-4885a29166ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:40:22.513750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:22.513728 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/61978b73-f8cf-4e4a-8314-4885a29166ff-maas-api-xks-ca" pod "61978b73-f8cf-4e4a-8314-4885a29166ff" (UID: "61978b73-f8cf-4e4a-8314-4885a29166ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:40:22.513864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:22.513844 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61978b73-f8cf-4e4a-8314-4885a29166ff-kube-api-access-rkhmp" pod "61978b73-f8cf-4e4a-8314-4885a29166ff" (UID: "61978b73-f8cf-4e4a-8314-4885a29166ff"). InnerVolumeSpecName "kube-api-access-rkhmp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:40:22.612468 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:22.612385 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/61978b73-f8cf-4e4a-8314-4885a29166ff-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:40:22.612468 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:22.612416 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rkhmp\" (UniqueName: \"kubernetes.io/projected/61978b73-f8cf-4e4a-8314-4885a29166ff-kube-api-access-rkhmp\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:40:22.612468 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:22.612426 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/61978b73-f8cf-4e4a-8314-4885a29166ff-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:40:23.295454 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:23.295412 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4" event={"ID":"61978b73-f8cf-4e4a-8314-4885a29166ff","Type":"ContainerDied","Data":"5b6222f81e4bc29facb699d31400d4dd0172c36bb3589041a787b552abca35d4"} Oct 30 18:40:23.295454 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:23.295462 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5b6222f81e4bc29facb699d31400d4dd0172c36bb3589041a787b552abca35d4" Oct 30 18:40:23.295727 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:23.295479 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4" Oct 30 18:40:27.033862 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:40:27.033824 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:40:35.154455 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.154425 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj"] Oct 30 18:40:35.154890 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.154860 5232 cpu_manager.go:414] "containerMap: removing container" podUID="61978b73-f8cf-4e4a-8314-4885a29166ff" containerName="revoke-keys" Oct 30 18:40:35.154890 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.154875 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61978b73-f8cf-4e4a-8314-4885a29166ff" containerName="revoke-keys" Oct 30 18:40:35.154975 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.154946 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="61978b73-f8cf-4e4a-8314-4885a29166ff" containerName="revoke-keys" Oct 30 18:40:35.157948 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.157931 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.172834 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.172809 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-f57fb7-openshift-default-dockercfg-fw4v6\"" Oct 30 18:40:35.187159 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.187130 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj"] Oct 30 18:40:35.224035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.224004 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-envoy\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.224035 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.224036 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istiod-ca-cert\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.224224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.224071 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-workload-certs\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.224224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.224152 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-token\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.224224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.224178 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tsx5\" (UniqueName: \"kubernetes.io/projected/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-kube-api-access-7tsx5\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.224339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.224238 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-credential-socket\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.224339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.224254 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-podinfo\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.224339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.224317 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-data\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.224487 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.224352 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-workload-socket\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.325584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.325552 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-envoy\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.325584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.325584 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istiod-ca-cert\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.325819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.325612 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-workload-certs\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.325819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.325642 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-token\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.325819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.325671 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tsx5\" (UniqueName: \"kubernetes.io/projected/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-kube-api-access-7tsx5\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.325819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.325727 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-credential-socket\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.325819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.325747 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-podinfo\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.325819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.325776 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-data\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.325819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.325805 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-workload-socket\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.326208 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.326154 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-credential-socket\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.326273 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.326239 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-workload-certs\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.326501 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.326270 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-workload-socket\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.326501 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.326323 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-data\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.326501 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.326395 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istiod-ca-cert\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.328073 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.328041 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-envoy\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.328420 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.328404 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-podinfo\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.345424 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.345399 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tsx5\" (UniqueName: \"kubernetes.io/projected/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-kube-api-access-7tsx5\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.345652 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.345637 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-token\") pod \"e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.471555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.471439 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:35.774326 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.774297 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj"] Oct 30 18:40:35.775727 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:40:35.775696 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a5e3f2b_026e_4871_89ce_7ac9ed0f629c.slice/crio-c3e9ed38b911ff04213fb18eb2bfa14b406c4ea1bc4e8ed3479463161a93d986 WatchSource:0}: Error finding container c3e9ed38b911ff04213fb18eb2bfa14b406c4ea1bc4e8ed3479463161a93d986: Status 404 returned error can't find the container with id c3e9ed38b911ff04213fb18eb2bfa14b406c4ea1bc4e8ed3479463161a93d986 Oct 30 18:40:35.777903 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.777870 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:40:35.778017 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.777953 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:40:35.778017 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:35.778000 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:40:36.362793 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:36.362757 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" event={"ID":"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c","Type":"ContainerStarted","Data":"9911ececf9901cae8da804db45c617ec5ed6abf2e8a34317b6988827504dbed8"} Oct 30 18:40:36.363229 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:36.362807 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" event={"ID":"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c","Type":"ContainerStarted","Data":"c3e9ed38b911ff04213fb18eb2bfa14b406c4ea1bc4e8ed3479463161a93d986"} Oct 30 18:40:36.417457 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:36.417406 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" podStartSLOduration=1.417392698 podStartE2EDuration="1.417392698s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:36.415644728 +0000 UTC m=+2335.732065049" watchObservedRunningTime="2026-10-30 18:40:36.417392698 +0000 UTC m=+2335.733813018" Oct 30 18:40:36.471818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:36.471788 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:36.477778 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:36.477750 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:37.368047 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:37.368016 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:37.368913 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:37.368895 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:40.129714 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.129678 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n"] Oct 30 18:40:40.133329 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.133298 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" Oct 30 18:40:40.163361 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.163330 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:40:40.163586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.163381 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-f57fb7-serving-cert\"" Oct 30 18:40:40.163586 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.163534 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpqx\"" Oct 30 18:40:40.166925 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.166896 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n"] Oct 30 18:40:40.279731 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.279695 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n\" (UID: \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" Oct 30 18:40:40.279907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.279746 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n86v4\" (UniqueName: \"kubernetes.io/projected/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-kube-api-access-n86v4\") pod \"maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n\" (UID: \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" Oct 30 18:40:40.279907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.279829 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n\" (UID: \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" Oct 30 18:40:40.279907 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.279853 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n\" (UID: \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" Oct 30 18:40:40.380672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.380573 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n\" (UID: \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" Oct 30 18:40:40.380672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.380613 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n\" (UID: \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" Oct 30 18:40:40.380935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.380684 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n\" (UID: \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" Oct 30 18:40:40.380935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.380723 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n86v4\" (UniqueName: \"kubernetes.io/projected/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-kube-api-access-n86v4\") pod \"maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n\" (UID: \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" Oct 30 18:40:40.381469 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.381442 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n\" (UID: \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" Oct 30 18:40:40.383300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.383277 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n\" (UID: \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" Oct 30 18:40:40.383453 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.383341 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-metrics-certs\") pod \"maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n\" (UID: \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" Oct 30 18:40:40.399583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.399562 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n86v4\" (UniqueName: \"kubernetes.io/projected/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-kube-api-access-n86v4\") pod \"maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n\" (UID: \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" Oct 30 18:40:40.446341 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.446309 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" Oct 30 18:40:40.744204 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.744174 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n"] Oct 30 18:40:40.746440 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:40:40.746415 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ff830f2_9fb1_4ce5_aefa_e964d929ec23.slice/crio-fb82cd6eefff9b9cb725d49ba51d5c5ad395f7b37fa7af8450e2755fdd90b98f WatchSource:0}: Error finding container fb82cd6eefff9b9cb725d49ba51d5c5ad395f7b37fa7af8450e2755fdd90b98f: Status 404 returned error can't find the container with id fb82cd6eefff9b9cb725d49ba51d5c5ad395f7b37fa7af8450e2755fdd90b98f Oct 30 18:40:40.747022 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.746948 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv"] Oct 30 18:40:40.747862 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.747844 5232 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:40:40.751739 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.751718 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" Oct 30 18:40:40.767351 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.767318 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-9bb3fbce-dockercfg-fh755\"" Oct 30 18:40:40.768037 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.768013 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-f57fb7\"" Oct 30 18:40:40.769705 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.769678 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv"] Oct 30 18:40:40.863317 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.863276 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw"] Oct 30 18:40:40.867808 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.867783 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" Oct 30 18:40:40.886002 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.885752 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw"] Oct 30 18:40:40.886497 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.886460 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5nqbd\" (UniqueName: \"kubernetes.io/projected/2cdf9ceb-4144-4694-a24f-b73ae0a06f96-kube-api-access-5nqbd\") pod \"payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv\" (UID: \"2cdf9ceb-4144-4694-a24f-b73ae0a06f96\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" Oct 30 18:40:40.886680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.886542 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cdf9ceb-4144-4694-a24f-b73ae0a06f96-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv\" (UID: \"2cdf9ceb-4144-4694-a24f-b73ae0a06f96\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" Oct 30 18:40:40.987102 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.987070 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m65x8\" (UniqueName: \"kubernetes.io/projected/99e4007d-cd9b-4950-a80b-0383274c80f3-kube-api-access-m65x8\") pod \"payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw\" (UID: \"99e4007d-cd9b-4950-a80b-0383274c80f3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" Oct 30 18:40:40.987316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.987121 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5nqbd\" (UniqueName: \"kubernetes.io/projected/2cdf9ceb-4144-4694-a24f-b73ae0a06f96-kube-api-access-5nqbd\") pod \"payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv\" (UID: \"2cdf9ceb-4144-4694-a24f-b73ae0a06f96\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" Oct 30 18:40:40.987316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.987154 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cdf9ceb-4144-4694-a24f-b73ae0a06f96-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv\" (UID: \"2cdf9ceb-4144-4694-a24f-b73ae0a06f96\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" Oct 30 18:40:40.987316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.987220 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e4007d-cd9b-4950-a80b-0383274c80f3-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw\" (UID: \"99e4007d-cd9b-4950-a80b-0383274c80f3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" Oct 30 18:40:40.987925 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:40.987907 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cdf9ceb-4144-4694-a24f-b73ae0a06f96-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv\" (UID: \"2cdf9ceb-4144-4694-a24f-b73ae0a06f96\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" Oct 30 18:40:41.009785 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:41.009722 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5nqbd\" (UniqueName: \"kubernetes.io/projected/2cdf9ceb-4144-4694-a24f-b73ae0a06f96-kube-api-access-5nqbd\") pod \"payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv\" (UID: \"2cdf9ceb-4144-4694-a24f-b73ae0a06f96\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" Oct 30 18:40:41.079300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:41.079271 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-9bb3fbce-dockercfg-fh755\"" Oct 30 18:40:41.086497 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:41.086482 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" Oct 30 18:40:41.088005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:41.087985 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m65x8\" (UniqueName: \"kubernetes.io/projected/99e4007d-cd9b-4950-a80b-0383274c80f3-kube-api-access-m65x8\") pod \"payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw\" (UID: \"99e4007d-cd9b-4950-a80b-0383274c80f3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" Oct 30 18:40:41.088065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:41.088019 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e4007d-cd9b-4950-a80b-0383274c80f3-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw\" (UID: \"99e4007d-cd9b-4950-a80b-0383274c80f3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" Oct 30 18:40:41.088650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:41.088632 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e4007d-cd9b-4950-a80b-0383274c80f3-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw\" (UID: \"99e4007d-cd9b-4950-a80b-0383274c80f3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" Oct 30 18:40:41.107153 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:41.107127 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m65x8\" (UniqueName: \"kubernetes.io/projected/99e4007d-cd9b-4950-a80b-0383274c80f3-kube-api-access-m65x8\") pod \"payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw\" (UID: \"99e4007d-cd9b-4950-a80b-0383274c80f3\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" Oct 30 18:40:41.189613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:41.189579 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" Oct 30 18:40:41.379925 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:41.377247 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv"] Oct 30 18:40:41.379925 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:40:41.379795 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cdf9ceb_4144_4694_a24f_b73ae0a06f96.slice/crio-c0bf24885c79e436e0d1ecf255c4d8218e349d560795377a5a580a395c57ef01 WatchSource:0}: Error finding container c0bf24885c79e436e0d1ecf255c4d8218e349d560795377a5a580a395c57ef01: Status 404 returned error can't find the container with id c0bf24885c79e436e0d1ecf255c4d8218e349d560795377a5a580a395c57ef01 Oct 30 18:40:41.389790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:41.389750 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" event={"ID":"2cdf9ceb-4144-4694-a24f-b73ae0a06f96","Type":"ContainerStarted","Data":"c0bf24885c79e436e0d1ecf255c4d8218e349d560795377a5a580a395c57ef01"} Oct 30 18:40:41.391570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:41.391543 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" event={"ID":"8ff830f2-9fb1-4ce5-aefa-e964d929ec23","Type":"ContainerStarted","Data":"bdc852df0232c6defae8e52c316e4a8bb022d99a7e0d448bb5e7be1187c4ea7e"} Oct 30 18:40:41.391706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:41.391601 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" event={"ID":"8ff830f2-9fb1-4ce5-aefa-e964d929ec23","Type":"ContainerStarted","Data":"fb82cd6eefff9b9cb725d49ba51d5c5ad395f7b37fa7af8450e2755fdd90b98f"} Oct 30 18:40:41.391706 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:41.391633 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" Oct 30 18:40:41.431325 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:41.431279 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" podStartSLOduration=0.990020277 podStartE2EDuration="1.431264685s" totalImagesPullingTime="441.244408ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:41.43048341 +0000 UTC m=+2340.746903730" watchObservedRunningTime="2026-10-30 18:40:41.431264685 +0000 UTC m=+2340.747685006" Oct 30 18:40:41.501571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:41.501546 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw"] Oct 30 18:40:41.503133 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:40:41.503108 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99e4007d_cd9b_4950_a80b_0383274c80f3.slice/crio-f54ccd16e50dfc40ee1ffd098da13b9a7b9a65cc65f0913de0b83ee54bb76b1f WatchSource:0}: Error finding container f54ccd16e50dfc40ee1ffd098da13b9a7b9a65cc65f0913de0b83ee54bb76b1f: Status 404 returned error can't find the container with id f54ccd16e50dfc40ee1ffd098da13b9a7b9a65cc65f0913de0b83ee54bb76b1f Oct 30 18:40:42.398552 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:42.398478 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" event={"ID":"2cdf9ceb-4144-4694-a24f-b73ae0a06f96","Type":"ContainerStarted","Data":"4a3ff18848b8199ba42add282dd0d170399cea37b333c33bfe01475664234a91"} Oct 30 18:40:42.399014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:42.398710 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" Oct 30 18:40:42.400301 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:42.400279 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" event={"ID":"99e4007d-cd9b-4950-a80b-0383274c80f3","Type":"ContainerStarted","Data":"2c3f1a8336394eb6d570670baff96eac11aaf2565f113c058c57afff03f25a25"} Oct 30 18:40:42.400385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:42.400317 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" event={"ID":"99e4007d-cd9b-4950-a80b-0383274c80f3","Type":"ContainerStarted","Data":"f54ccd16e50dfc40ee1ffd098da13b9a7b9a65cc65f0913de0b83ee54bb76b1f"} Oct 30 18:40:42.400385 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:42.400364 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" Oct 30 18:40:42.438401 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:42.438356 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" podStartSLOduration=2.438344647 podStartE2EDuration="2.438344647s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:42.436549636 +0000 UTC m=+2341.752969956" watchObservedRunningTime="2026-10-30 18:40:42.438344647 +0000 UTC m=+2341.754764968" Oct 30 18:40:42.475930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:42.475888 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" podStartSLOduration=2.475875456 podStartE2EDuration="2.475875456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:42.474067912 +0000 UTC m=+2341.790488232" watchObservedRunningTime="2026-10-30 18:40:42.475875456 +0000 UTC m=+2341.792295774" Oct 30 18:40:47.406374 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:47.406342 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" Oct 30 18:40:53.369002 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:53.368955 5232 patch_prober.go:29] interesting pod/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:40:53.369462 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:53.369038 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" podUID="4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:40:53.409304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:53.409277 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" Oct 30 18:40:53.409458 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:53.409351 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" Oct 30 18:40:54.485914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:54.485878 5232 generic.go:356] "Generic (PLEG): container finished" podID="4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" containerID="9911ececf9901cae8da804db45c617ec5ed6abf2e8a34317b6988827504dbed8" exitCode=0 Oct 30 18:40:54.486416 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:54.485954 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" event={"ID":"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c","Type":"ContainerDied","Data":"9911ececf9901cae8da804db45c617ec5ed6abf2e8a34317b6988827504dbed8"} Oct 30 18:40:54.486416 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:54.486340 5232 scope.go:122] "RemoveContainer" containerID="9911ececf9901cae8da804db45c617ec5ed6abf2e8a34317b6988827504dbed8" Oct 30 18:40:54.487546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:54.487495 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:40:54.487627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:54.487586 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:40:54.487668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:54.487628 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:40:55.493306 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:55.493267 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" event={"ID":"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c","Type":"ContainerStarted","Data":"ae2c59c5478461f2cb720dcc4fa7cf3cd8f2bc52188833f40986d63793698948"} Oct 30 18:40:55.607601 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:55.607561 5232 scope.go:122] "RemoveContainer" containerID="7e0d40bf7f7b72d08bc908f1a74172d7f9f019eef60f39c1eda9c8d121a090a1" Oct 30 18:40:55.640196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:55.640168 5232 scope.go:122] "RemoveContainer" containerID="79a9432425ed220d73e1fcf831e5578090442da4f838505f3036ebb321e6c641" Oct 30 18:40:56.038536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.029058 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz"] Oct 30 18:40:56.041885 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.041820 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.043940 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.043915 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e774cd-d21a-457a-9d24-f7ad18daeedb-tls-certs\") pod \"simulator-single-main-f57fb7-kserve-f69458797-22rnz\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.044152 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.044133 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-model-cache\") pod \"simulator-single-main-f57fb7-kserve-f69458797-22rnz\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.044295 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.044278 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgw52\" (UniqueName: \"kubernetes.io/projected/d6e774cd-d21a-457a-9d24-f7ad18daeedb-kube-api-access-tgw52\") pod \"simulator-single-main-f57fb7-kserve-f69458797-22rnz\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.044418 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.044403 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-tmp-dir\") pod \"simulator-single-main-f57fb7-kserve-f69458797-22rnz\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.044604 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.044587 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-dshm\") pod \"simulator-single-main-f57fb7-kserve-f69458797-22rnz\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.044744 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.044731 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-home\") pod \"simulator-single-main-f57fb7-kserve-f69458797-22rnz\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.044876 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.044334 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz"] Oct 30 18:40:56.073685 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.073654 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f57fb7\"/\"kube-root-ca.crt\"" Oct 30 18:40:56.074572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.073991 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-f57fb7\"/\"openshift-service-ca.crt\"" Oct 30 18:40:56.074572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.074065 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f57fb7\"/\"simulator-single-main-f57fb7-kserve-self-signed-certs\"" Oct 30 18:40:56.074572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.074122 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-f57fb7\"/\"default-dockercfg-gs2z8\"" Oct 30 18:40:56.145332 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.145297 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-dshm\") pod \"simulator-single-main-f57fb7-kserve-f69458797-22rnz\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.145332 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.145341 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-home\") pod \"simulator-single-main-f57fb7-kserve-f69458797-22rnz\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.145612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.145374 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e774cd-d21a-457a-9d24-f7ad18daeedb-tls-certs\") pod \"simulator-single-main-f57fb7-kserve-f69458797-22rnz\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.145612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.145392 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-model-cache\") pod \"simulator-single-main-f57fb7-kserve-f69458797-22rnz\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.145612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.145421 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgw52\" (UniqueName: \"kubernetes.io/projected/d6e774cd-d21a-457a-9d24-f7ad18daeedb-kube-api-access-tgw52\") pod \"simulator-single-main-f57fb7-kserve-f69458797-22rnz\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.145612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.145438 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-tmp-dir\") pod \"simulator-single-main-f57fb7-kserve-f69458797-22rnz\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.145926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.145897 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-tmp-dir\") pod \"simulator-single-main-f57fb7-kserve-f69458797-22rnz\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.146116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.146088 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-home\") pod \"simulator-single-main-f57fb7-kserve-f69458797-22rnz\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.146266 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.146242 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-model-cache\") pod \"simulator-single-main-f57fb7-kserve-f69458797-22rnz\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.148239 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.148215 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e774cd-d21a-457a-9d24-f7ad18daeedb-tls-certs\") pod \"simulator-single-main-f57fb7-kserve-f69458797-22rnz\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.148759 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.148704 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-dshm\") pod \"simulator-single-main-f57fb7-kserve-f69458797-22rnz\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.164875 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.164852 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgw52\" (UniqueName: \"kubernetes.io/projected/d6e774cd-d21a-457a-9d24-f7ad18daeedb-kube-api-access-tgw52\") pod \"simulator-single-main-f57fb7-kserve-f69458797-22rnz\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.369070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.368986 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:56.472055 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.472020 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:40:56.472278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.472252 5232 patch_prober.go:29] interesting pod/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body= Oct 30 18:40:56.472332 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.472304 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" podUID="4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" Oct 30 18:40:56.684041 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:56.684013 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz"] Oct 30 18:40:56.685264 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:40:56.685237 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd6e774cd_d21a_457a_9d24_f7ad18daeedb.slice/crio-d84be7d73db560bd07571eeab7d48e84e071e1ff24bee453ae57ce9f547bb144 WatchSource:0}: Error finding container d84be7d73db560bd07571eeab7d48e84e071e1ff24bee453ae57ce9f547bb144: Status 404 returned error can't find the container with id d84be7d73db560bd07571eeab7d48e84e071e1ff24bee453ae57ce9f547bb144 Oct 30 18:40:57.057104 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:40:57.057059 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:40:57.472422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:57.472335 5232 patch_prober.go:29] interesting pod/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body= Oct 30 18:40:57.472422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:57.472390 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" podUID="4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" Oct 30 18:40:57.506707 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:57.506670 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" event={"ID":"d6e774cd-d21a-457a-9d24-f7ad18daeedb","Type":"ContainerStarted","Data":"01492e2967347b923aa53e3321c2f81862d0af681a97a2590704582a762ae623"} Oct 30 18:40:57.506707 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:57.506711 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" event={"ID":"d6e774cd-d21a-457a-9d24-f7ad18daeedb","Type":"ContainerStarted","Data":"d84be7d73db560bd07571eeab7d48e84e071e1ff24bee453ae57ce9f547bb144"} Oct 30 18:40:57.550714 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:57.550659 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" podStartSLOduration=2.550646098 podStartE2EDuration="2.550646098s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:40:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:40:57.549319673 +0000 UTC m=+2356.865739995" watchObservedRunningTime="2026-10-30 18:40:57.550646098 +0000 UTC m=+2356.867066419" Oct 30 18:40:58.034549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:58.034337 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5"] Oct 30 18:40:58.037734 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:58.037707 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c1e271a9-dvfw5"] Oct 30 18:40:58.370174 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:58.370093 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:40:58.471850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:58.471821 5232 patch_prober.go:29] interesting pod/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body= Oct 30 18:40:58.472033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:58.471872 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" podUID="4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" Oct 30 18:40:58.800976 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:58.800943 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b32b717-279b-4232-a275-d6ac72d86937" path="/var/lib/kubelet/pods/8b32b717-279b-4232-a275-d6ac72d86937/volumes" Oct 30 18:40:59.472059 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:59.472019 5232 patch_prober.go:29] interesting pod/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" start-of-body= Oct 30 18:40:59.472574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:40:59.472075 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" podUID="4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.8:15021/healthz/ready\": dial tcp 10.133.1.8:15021: connect: connection refused" Oct 30 18:41:00.928331 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:00.928292 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:41:00.928887 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:00.928835 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:41:00.930127 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:00.930104 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:41:06.370024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:06.369988 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:41:06.382497 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:06.382471 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:41:06.588040 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:06.588003 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:41:24.026526 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:24.026480 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp"] Oct 30 18:41:24.037591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:24.037552 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-g8pzp"] Oct 30 18:41:24.801359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:24.801321 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8aee1942-38a1-42af-9826-c316b3363258" path="/var/lib/kubelet/pods/8aee1942-38a1-42af-9826-c316b3363258/volumes" Oct 30 18:41:27.032226 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:41:27.032191 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:41:36.031115 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:36.031074 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq"] Oct 30 18:41:36.040472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:36.040439 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-w5mqq"] Oct 30 18:41:36.800989 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:36.800956 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad05b971-8c8d-4dd8-ab52-e402486ca16a" path="/var/lib/kubelet/pods/ad05b971-8c8d-4dd8-ab52-e402486ca16a/volumes" Oct 30 18:41:44.035145 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:44.035102 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz"] Oct 30 18:41:49.748762 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:49.748725 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw"] Oct 30 18:41:49.784015 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:49.783983 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv"] Oct 30 18:41:51.719518 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:51.719479 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n"] Oct 30 18:41:51.720023 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:51.719755 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" podUID="8ff830f2-9fb1-4ce5-aefa-e964d929ec23" containerName="maas-api" containerID="cri-o://bdc852df0232c6defae8e52c316e4a8bb022d99a7e0d448bb5e7be1187c4ea7e" gracePeriod=30 Oct 30 18:41:52.182548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.182500 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" Oct 30 18:41:52.272622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.272502 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-metrics-certs\") pod \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\" (UID: \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\") " Oct 30 18:41:52.272622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.272568 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-openshift-service-ca\") pod \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\" (UID: \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\") " Oct 30 18:41:52.272879 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.272647 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-maas-api-tls\") pod \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\" (UID: \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\") " Oct 30 18:41:52.272879 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.272703 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-kube-api-access-n86v4\" (UniqueName: \"kubernetes.io/projected/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-kube-api-access-n86v4\") pod \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\" (UID: \"8ff830f2-9fb1-4ce5-aefa-e964d929ec23\") " Oct 30 18:41:52.272992 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.272947 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-openshift-service-ca" pod "8ff830f2-9fb1-4ce5-aefa-e964d929ec23" (UID: "8ff830f2-9fb1-4ce5-aefa-e964d929ec23"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:41:52.274908 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.274877 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-kube-api-access-n86v4" pod "8ff830f2-9fb1-4ce5-aefa-e964d929ec23" (UID: "8ff830f2-9fb1-4ce5-aefa-e964d929ec23"). InnerVolumeSpecName "kube-api-access-n86v4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:52.275062 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.274915 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-maas-api-tls" pod "8ff830f2-9fb1-4ce5-aefa-e964d929ec23" (UID: "8ff830f2-9fb1-4ce5-aefa-e964d929ec23"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:52.275062 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.274937 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-metrics-certs" pod "8ff830f2-9fb1-4ce5-aefa-e964d929ec23" (UID: "8ff830f2-9fb1-4ce5-aefa-e964d929ec23"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:52.374313 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.374279 5232 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-metrics-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:41:52.374313 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.374307 5232 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-openshift-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:41:52.374313 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.374316 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-maas-api-tls\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:41:52.374591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.374325 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n86v4\" (UniqueName: \"kubernetes.io/projected/8ff830f2-9fb1-4ce5-aefa-e964d929ec23-kube-api-access-n86v4\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:41:52.803673 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.803641 5232 generic.go:356] "Generic (PLEG): container finished" podID="8ff830f2-9fb1-4ce5-aefa-e964d929ec23" containerID="bdc852df0232c6defae8e52c316e4a8bb022d99a7e0d448bb5e7be1187c4ea7e" exitCode=0 Oct 30 18:41:52.804258 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.803687 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" Oct 30 18:41:52.804258 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.803728 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" event={"ID":"8ff830f2-9fb1-4ce5-aefa-e964d929ec23","Type":"ContainerDied","Data":"bdc852df0232c6defae8e52c316e4a8bb022d99a7e0d448bb5e7be1187c4ea7e"} Oct 30 18:41:52.804258 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.803837 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n" event={"ID":"8ff830f2-9fb1-4ce5-aefa-e964d929ec23","Type":"ContainerDied","Data":"fb82cd6eefff9b9cb725d49ba51d5c5ad395f7b37fa7af8450e2755fdd90b98f"} Oct 30 18:41:52.804258 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.803858 5232 scope.go:122] "RemoveContainer" containerID="bdc852df0232c6defae8e52c316e4a8bb022d99a7e0d448bb5e7be1187c4ea7e" Oct 30 18:41:52.814565 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.814547 5232 scope.go:122] "RemoveContainer" containerID="bdc852df0232c6defae8e52c316e4a8bb022d99a7e0d448bb5e7be1187c4ea7e" Oct 30 18:41:52.814864 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:41:52.814846 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bdc852df0232c6defae8e52c316e4a8bb022d99a7e0d448bb5e7be1187c4ea7e\": container with ID starting with bdc852df0232c6defae8e52c316e4a8bb022d99a7e0d448bb5e7be1187c4ea7e not found: ID does not exist" containerID="bdc852df0232c6defae8e52c316e4a8bb022d99a7e0d448bb5e7be1187c4ea7e" Oct 30 18:41:52.814942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.814875 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bdc852df0232c6defae8e52c316e4a8bb022d99a7e0d448bb5e7be1187c4ea7e"} err="failed to get container status \"bdc852df0232c6defae8e52c316e4a8bb022d99a7e0d448bb5e7be1187c4ea7e\": rpc error: code = NotFound desc = could not find container \"bdc852df0232c6defae8e52c316e4a8bb022d99a7e0d448bb5e7be1187c4ea7e\": container with ID starting with bdc852df0232c6defae8e52c316e4a8bb022d99a7e0d448bb5e7be1187c4ea7e not found: ID does not exist" Oct 30 18:41:52.856603 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.856576 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n"] Oct 30 18:41:52.869081 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:52.869056 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-f57fb7-67fc77f6fc-qsz4n"] Oct 30 18:41:54.749065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:54.749004 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" podUID="99e4007d-cd9b-4950-a80b-0383274c80f3" containerName="payload-processing" containerID="cri-o://2c3f1a8336394eb6d570670baff96eac11aaf2565f113c058c57afff03f25a25" gracePeriod=25 Oct 30 18:41:54.784433 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:54.784394 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" podUID="2cdf9ceb-4144-4694-a24f-b73ae0a06f96" containerName="payload-pre-processing" containerID="cri-o://4a3ff18848b8199ba42add282dd0d170399cea37b333c33bfe01475664234a91" gracePeriod=25 Oct 30 18:41:54.807564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:54.807532 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ff830f2-9fb1-4ce5-aefa-e964d929ec23" path="/var/lib/kubelet/pods/8ff830f2-9fb1-4ce5-aefa-e964d929ec23/volumes" Oct 30 18:41:55.664447 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:55.664412 5232 scope.go:122] "RemoveContainer" containerID="de457d1690f24c25ca03aaea246614753f3bd07773c0bd8e8a25cb14e0537fae" Oct 30 18:41:57.033409 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:41:57.033365 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:41:59.047171 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.047133 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj"] Oct 30 18:41:59.047670 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.047368 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" podUID="4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" containerName="istio-proxy" containerID="cri-o://ae2c59c5478461f2cb720dcc4fa7cf3cd8f2bc52188833f40986d63793698948" gracePeriod=30 Oct 30 18:41:59.055241 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.055205 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" podUID="d6e774cd-d21a-457a-9d24-f7ad18daeedb" containerName="main" containerID="cri-o://01492e2967347b923aa53e3321c2f81862d0af681a97a2590704582a762ae623" gracePeriod=45 Oct 30 18:41:59.310775 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.310718 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:41:59.446724 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.446691 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-tmp-dir\") pod \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " Oct 30 18:41:59.446913 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.446741 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-model-cache\") pod \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " Oct 30 18:41:59.446913 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.446874 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6e774cd-d21a-457a-9d24-f7ad18daeedb-kube-api-access-tgw52\" (UniqueName: \"kubernetes.io/projected/d6e774cd-d21a-457a-9d24-f7ad18daeedb-kube-api-access-tgw52\") pod \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " Oct 30 18:41:59.446913 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.446893 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-home\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-home\") pod \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " Oct 30 18:41:59.447095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.446916 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d6e774cd-d21a-457a-9d24-f7ad18daeedb-tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e774cd-d21a-457a-9d24-f7ad18daeedb-tls-certs\") pod \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " Oct 30 18:41:59.447095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.446935 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-dshm\") pod \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\" (UID: \"d6e774cd-d21a-457a-9d24-f7ad18daeedb\") " Oct 30 18:41:59.447095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.447024 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-model-cache" pod "d6e774cd-d21a-457a-9d24-f7ad18daeedb" (UID: "d6e774cd-d21a-457a-9d24-f7ad18daeedb"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:59.447095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.447031 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-tmp-dir" pod "d6e774cd-d21a-457a-9d24-f7ad18daeedb" (UID: "d6e774cd-d21a-457a-9d24-f7ad18daeedb"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:59.447304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.447191 5232 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-tmp-dir\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:41:59.447304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.447189 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-home" pod "d6e774cd-d21a-457a-9d24-f7ad18daeedb" (UID: "d6e774cd-d21a-457a-9d24-f7ad18daeedb"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:59.447304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.447205 5232 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-model-cache\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:41:59.449311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.449280 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-dshm" pod "d6e774cd-d21a-457a-9d24-f7ad18daeedb" (UID: "d6e774cd-d21a-457a-9d24-f7ad18daeedb"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:41:59.449311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.449288 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d6e774cd-d21a-457a-9d24-f7ad18daeedb-tls-certs" pod "d6e774cd-d21a-457a-9d24-f7ad18daeedb" (UID: "d6e774cd-d21a-457a-9d24-f7ad18daeedb"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:41:59.449311 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.449280 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6e774cd-d21a-457a-9d24-f7ad18daeedb-kube-api-access-tgw52" pod "d6e774cd-d21a-457a-9d24-f7ad18daeedb" (UID: "d6e774cd-d21a-457a-9d24-f7ad18daeedb"). InnerVolumeSpecName "kube-api-access-tgw52". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:41:59.547792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.547756 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgw52\" (UniqueName: \"kubernetes.io/projected/d6e774cd-d21a-457a-9d24-f7ad18daeedb-kube-api-access-tgw52\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:41:59.547792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.547790 5232 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-home\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:41:59.547792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.547801 5232 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/d6e774cd-d21a-457a-9d24-f7ad18daeedb-tls-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:41:59.548057 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.547810 5232 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/d6e774cd-d21a-457a-9d24-f7ad18daeedb-dshm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:41:59.839095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.839062 5232 generic.go:356] "Generic (PLEG): container finished" podID="d6e774cd-d21a-457a-9d24-f7ad18daeedb" containerID="01492e2967347b923aa53e3321c2f81862d0af681a97a2590704582a762ae623" exitCode=0 Oct 30 18:41:59.839300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.839111 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" Oct 30 18:41:59.839300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.839127 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" event={"ID":"d6e774cd-d21a-457a-9d24-f7ad18daeedb","Type":"ContainerDied","Data":"01492e2967347b923aa53e3321c2f81862d0af681a97a2590704582a762ae623"} Oct 30 18:41:59.839300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.839159 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz" event={"ID":"d6e774cd-d21a-457a-9d24-f7ad18daeedb","Type":"ContainerDied","Data":"d84be7d73db560bd07571eeab7d48e84e071e1ff24bee453ae57ce9f547bb144"} Oct 30 18:41:59.839300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.839175 5232 scope.go:122] "RemoveContainer" containerID="01492e2967347b923aa53e3321c2f81862d0af681a97a2590704582a762ae623" Oct 30 18:41:59.848661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.848620 5232 scope.go:122] "RemoveContainer" containerID="01492e2967347b923aa53e3321c2f81862d0af681a97a2590704582a762ae623" Oct 30 18:41:59.848902 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:41:59.848884 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"01492e2967347b923aa53e3321c2f81862d0af681a97a2590704582a762ae623\": container with ID starting with 01492e2967347b923aa53e3321c2f81862d0af681a97a2590704582a762ae623 not found: ID does not exist" containerID="01492e2967347b923aa53e3321c2f81862d0af681a97a2590704582a762ae623" Oct 30 18:41:59.848950 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.848910 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"01492e2967347b923aa53e3321c2f81862d0af681a97a2590704582a762ae623"} err="failed to get container status \"01492e2967347b923aa53e3321c2f81862d0af681a97a2590704582a762ae623\": rpc error: code = NotFound desc = could not find container \"01492e2967347b923aa53e3321c2f81862d0af681a97a2590704582a762ae623\": container with ID starting with 01492e2967347b923aa53e3321c2f81862d0af681a97a2590704582a762ae623 not found: ID does not exist" Oct 30 18:41:59.888913 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.888878 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz"] Oct 30 18:41:59.901083 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:41:59.901055 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-f57fb7/simulator-single-main-f57fb7-kserve-f69458797-22rnz"] Oct 30 18:42:00.336820 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.336783 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4"] Oct 30 18:42:00.339004 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.338976 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-b9wk4"] Oct 30 18:42:00.379491 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.379462 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp"] Oct 30 18:42:00.379885 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.379871 5232 cpu_manager.go:414] "containerMap: removing container" podUID="8ff830f2-9fb1-4ce5-aefa-e964d929ec23" containerName="maas-api" Oct 30 18:42:00.379938 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.379887 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ff830f2-9fb1-4ce5-aefa-e964d929ec23" containerName="maas-api" Oct 30 18:42:00.379938 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.379899 5232 cpu_manager.go:414] "containerMap: removing container" podUID="d6e774cd-d21a-457a-9d24-f7ad18daeedb" containerName="main" Oct 30 18:42:00.379938 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.379904 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6e774cd-d21a-457a-9d24-f7ad18daeedb" containerName="main" Oct 30 18:42:00.380043 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.379988 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ff830f2-9fb1-4ce5-aefa-e964d929ec23" containerName="maas-api" Oct 30 18:42:00.380043 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.379997 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6e774cd-d21a-457a-9d24-f7ad18daeedb" containerName="main" Oct 30 18:42:00.384562 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.384542 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp" Oct 30 18:42:00.400022 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.399993 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:42:00.400022 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.399996 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:42:00.401699 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.401678 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp"] Oct 30 18:42:00.456718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.456690 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7044133a-08fc-4861-b938-941ea0a59b2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qsknp\" (UID: \"7044133a-08fc-4861-b938-941ea0a59b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp" Oct 30 18:42:00.456888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.456750 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7044133a-08fc-4861-b938-941ea0a59b2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qsknp\" (UID: \"7044133a-08fc-4861-b938-941ea0a59b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp" Oct 30 18:42:00.456888 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.456830 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4cmg\" (UniqueName: \"kubernetes.io/projected/7044133a-08fc-4861-b938-941ea0a59b2d-kube-api-access-x4cmg\") pod \"maas-api-revoke-sub-3d4156f1-qsknp\" (UID: \"7044133a-08fc-4861-b938-941ea0a59b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp" Oct 30 18:42:00.558912 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.558880 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7044133a-08fc-4861-b938-941ea0a59b2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qsknp\" (UID: \"7044133a-08fc-4861-b938-941ea0a59b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp" Oct 30 18:42:00.559070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.558939 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7044133a-08fc-4861-b938-941ea0a59b2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qsknp\" (UID: \"7044133a-08fc-4861-b938-941ea0a59b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp" Oct 30 18:42:00.559070 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.558972 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4cmg\" (UniqueName: \"kubernetes.io/projected/7044133a-08fc-4861-b938-941ea0a59b2d-kube-api-access-x4cmg\") pod \"maas-api-revoke-sub-3d4156f1-qsknp\" (UID: \"7044133a-08fc-4861-b938-941ea0a59b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp" Oct 30 18:42:00.559779 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.559713 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7044133a-08fc-4861-b938-941ea0a59b2d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-qsknp\" (UID: \"7044133a-08fc-4861-b938-941ea0a59b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp" Oct 30 18:42:00.561475 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.561448 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7044133a-08fc-4861-b938-941ea0a59b2d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-qsknp\" (UID: \"7044133a-08fc-4861-b938-941ea0a59b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp" Oct 30 18:42:00.577793 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.577764 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4cmg\" (UniqueName: \"kubernetes.io/projected/7044133a-08fc-4861-b938-941ea0a59b2d-kube-api-access-x4cmg\") pod \"maas-api-revoke-sub-3d4156f1-qsknp\" (UID: \"7044133a-08fc-4861-b938-941ea0a59b2d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp" Oct 30 18:42:00.697318 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.697234 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp" Oct 30 18:42:00.802483 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.802448 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61978b73-f8cf-4e4a-8314-4885a29166ff" path="/var/lib/kubelet/pods/61978b73-f8cf-4e4a-8314-4885a29166ff/volumes" Oct 30 18:42:00.802885 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.802871 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6e774cd-d21a-457a-9d24-f7ad18daeedb" path="/var/lib/kubelet/pods/d6e774cd-d21a-457a-9d24-f7ad18daeedb/volumes" Oct 30 18:42:00.931281 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.931246 5232 patch_prober.go:29] interesting pod/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:42:00.931444 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.931303 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" podUID="4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:42:00.986612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:00.986586 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp"] Oct 30 18:42:00.988102 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:42:00.988076 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7044133a_08fc_4861_b938_941ea0a59b2d.slice/crio-d5d9fadd87e02a2079f8d37d74e3155e7bb7b0ba2d72c9c93cb4057a0c3f0e4b WatchSource:0}: Error finding container d5d9fadd87e02a2079f8d37d74e3155e7bb7b0ba2d72c9c93cb4057a0c3f0e4b: Status 404 returned error can't find the container with id d5d9fadd87e02a2079f8d37d74e3155e7bb7b0ba2d72c9c93cb4057a0c3f0e4b Oct 30 18:42:01.852193 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:01.852164 5232 generic.go:356] "Generic (PLEG): container finished" podID="7044133a-08fc-4861-b938-941ea0a59b2d" containerID="a29b043ed2db931533c4624d00d2d7665761e33a0e60172554a1d9c7bc4c824e" exitCode=0 Oct 30 18:42:01.852581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:01.852231 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp" event={"ID":"7044133a-08fc-4861-b938-941ea0a59b2d","Type":"ContainerDied","Data":"a29b043ed2db931533c4624d00d2d7665761e33a0e60172554a1d9c7bc4c824e"} Oct 30 18:42:01.852581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:01.852267 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp" event={"ID":"7044133a-08fc-4861-b938-941ea0a59b2d","Type":"ContainerStarted","Data":"d5d9fadd87e02a2079f8d37d74e3155e7bb7b0ba2d72c9c93cb4057a0c3f0e4b"} Oct 30 18:42:02.990045 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:02.990021 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp" Oct 30 18:42:03.083814 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:03.083784 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7044133a-08fc-4861-b938-941ea0a59b2d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7044133a-08fc-4861-b938-941ea0a59b2d-maas-api-openshift-ca\") pod \"7044133a-08fc-4861-b938-941ea0a59b2d\" (UID: \"7044133a-08fc-4861-b938-941ea0a59b2d\") " Oct 30 18:42:03.083969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:03.083877 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7044133a-08fc-4861-b938-941ea0a59b2d-kube-api-access-x4cmg\" (UniqueName: \"kubernetes.io/projected/7044133a-08fc-4861-b938-941ea0a59b2d-kube-api-access-x4cmg\") pod \"7044133a-08fc-4861-b938-941ea0a59b2d\" (UID: \"7044133a-08fc-4861-b938-941ea0a59b2d\") " Oct 30 18:42:03.083969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:03.083946 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7044133a-08fc-4861-b938-941ea0a59b2d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7044133a-08fc-4861-b938-941ea0a59b2d-maas-api-xks-ca\") pod \"7044133a-08fc-4861-b938-941ea0a59b2d\" (UID: \"7044133a-08fc-4861-b938-941ea0a59b2d\") " Oct 30 18:42:03.084189 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:03.084161 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7044133a-08fc-4861-b938-941ea0a59b2d-maas-api-openshift-ca" pod "7044133a-08fc-4861-b938-941ea0a59b2d" (UID: "7044133a-08fc-4861-b938-941ea0a59b2d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:03.086183 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:03.086160 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7044133a-08fc-4861-b938-941ea0a59b2d-kube-api-access-x4cmg" pod "7044133a-08fc-4861-b938-941ea0a59b2d" (UID: "7044133a-08fc-4861-b938-941ea0a59b2d"). InnerVolumeSpecName "kube-api-access-x4cmg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:03.086278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:03.086257 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7044133a-08fc-4861-b938-941ea0a59b2d-maas-api-xks-ca" pod "7044133a-08fc-4861-b938-941ea0a59b2d" (UID: "7044133a-08fc-4861-b938-941ea0a59b2d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:42:03.184746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:03.184683 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4cmg\" (UniqueName: \"kubernetes.io/projected/7044133a-08fc-4861-b938-941ea0a59b2d-kube-api-access-x4cmg\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:42:03.184746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:03.184708 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7044133a-08fc-4861-b938-941ea0a59b2d-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:42:03.184746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:03.184718 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7044133a-08fc-4861-b938-941ea0a59b2d-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:42:03.407245 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:03.407207 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" podUID="2cdf9ceb-4144-4694-a24f-b73ae0a06f96" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:42:03.407421 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:03.407305 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" podUID="99e4007d-cd9b-4950-a80b-0383274c80f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:42:03.867394 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:03.867365 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp" Oct 30 18:42:03.867627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:03.867410 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp" event={"ID":"7044133a-08fc-4861-b938-941ea0a59b2d","Type":"ContainerDied","Data":"d5d9fadd87e02a2079f8d37d74e3155e7bb7b0ba2d72c9c93cb4057a0c3f0e4b"} Oct 30 18:42:03.867627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:03.867466 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5d9fadd87e02a2079f8d37d74e3155e7bb7b0ba2d72c9c93cb4057a0c3f0e4b" Oct 30 18:42:04.299641 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.299614 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:42:04.396960 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.396931 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-envoy\") pod \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " Oct 30 18:42:04.397122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.396981 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-data\") pod \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " Oct 30 18:42:04.397122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.397017 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-kube-api-access-7tsx5\" (UniqueName: \"kubernetes.io/projected/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-kube-api-access-7tsx5\") pod \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " Oct 30 18:42:04.397122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.397057 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-token\" (UniqueName: \"kubernetes.io/projected/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-token\") pod \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " Oct 30 18:42:04.397122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.397086 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-credential-socket\") pod \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " Oct 30 18:42:04.397863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.397135 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-workload-certs\") pod \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " Oct 30 18:42:04.397863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.397179 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istiod-ca-cert\") pod \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " Oct 30 18:42:04.397863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.397211 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-workload-socket\") pod \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " Oct 30 18:42:04.397863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.397251 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-podinfo\") pod \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\" (UID: \"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c\") " Oct 30 18:42:04.397863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.397277 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-data" pod "4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" (UID: "4a5e3f2b-026e-4871-89ce-7ac9ed0f629c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:04.397863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.397567 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-credential-socket" pod "4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" (UID: "4a5e3f2b-026e-4871-89ce-7ac9ed0f629c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:04.397863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.397602 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:42:04.397863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.397653 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-workload-socket" pod "4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" (UID: "4a5e3f2b-026e-4871-89ce-7ac9ed0f629c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:04.397863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.397710 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istiod-ca-cert" pod "4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" (UID: "4a5e3f2b-026e-4871-89ce-7ac9ed0f629c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:04.397863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.397755 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-workload-certs" pod "4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" (UID: "4a5e3f2b-026e-4871-89ce-7ac9ed0f629c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:04.399535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.399482 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-envoy" pod "4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" (UID: "4a5e3f2b-026e-4871-89ce-7ac9ed0f629c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:42:04.399535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.399491 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-token" pod "4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" (UID: "4a5e3f2b-026e-4871-89ce-7ac9ed0f629c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:04.399722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.399537 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-kube-api-access-7tsx5" pod "4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" (UID: "4a5e3f2b-026e-4871-89ce-7ac9ed0f629c"). InnerVolumeSpecName "kube-api-access-7tsx5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:04.399838 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.399820 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-podinfo" pod "4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" (UID: "4a5e3f2b-026e-4871-89ce-7ac9ed0f629c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:42:04.498060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.498033 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:42:04.498060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.498059 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:42:04.498240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.498068 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:42:04.498240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.498079 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:42:04.498240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.498088 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:42:04.498240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.498096 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tsx5\" (UniqueName: \"kubernetes.io/projected/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-kube-api-access-7tsx5\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:42:04.498240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.498105 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:42:04.498240 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.498113 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:42:04.873753 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.873704 5232 generic.go:356] "Generic (PLEG): container finished" podID="4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" containerID="ae2c59c5478461f2cb720dcc4fa7cf3cd8f2bc52188833f40986d63793698948" exitCode=0 Oct 30 18:42:04.873981 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.873761 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" Oct 30 18:42:04.873981 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.873773 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" event={"ID":"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c","Type":"ContainerDied","Data":"ae2c59c5478461f2cb720dcc4fa7cf3cd8f2bc52188833f40986d63793698948"} Oct 30 18:42:04.873981 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.873831 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj" event={"ID":"4a5e3f2b-026e-4871-89ce-7ac9ed0f629c","Type":"ContainerDied","Data":"c3e9ed38b911ff04213fb18eb2bfa14b406c4ea1bc4e8ed3479463161a93d986"} Oct 30 18:42:04.873981 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.873860 5232 scope.go:122] "RemoveContainer" containerID="ae2c59c5478461f2cb720dcc4fa7cf3cd8f2bc52188833f40986d63793698948" Oct 30 18:42:04.882842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.882820 5232 scope.go:122] "RemoveContainer" containerID="9911ececf9901cae8da804db45c617ec5ed6abf2e8a34317b6988827504dbed8" Oct 30 18:42:04.891281 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.891265 5232 scope.go:122] "RemoveContainer" containerID="ae2c59c5478461f2cb720dcc4fa7cf3cd8f2bc52188833f40986d63793698948" Oct 30 18:42:04.891565 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:42:04.891544 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae2c59c5478461f2cb720dcc4fa7cf3cd8f2bc52188833f40986d63793698948\": container with ID starting with ae2c59c5478461f2cb720dcc4fa7cf3cd8f2bc52188833f40986d63793698948 not found: ID does not exist" containerID="ae2c59c5478461f2cb720dcc4fa7cf3cd8f2bc52188833f40986d63793698948" Oct 30 18:42:04.891626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.891574 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae2c59c5478461f2cb720dcc4fa7cf3cd8f2bc52188833f40986d63793698948"} err="failed to get container status \"ae2c59c5478461f2cb720dcc4fa7cf3cd8f2bc52188833f40986d63793698948\": rpc error: code = NotFound desc = could not find container \"ae2c59c5478461f2cb720dcc4fa7cf3cd8f2bc52188833f40986d63793698948\": container with ID starting with ae2c59c5478461f2cb720dcc4fa7cf3cd8f2bc52188833f40986d63793698948 not found: ID does not exist" Oct 30 18:42:04.891626 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.891590 5232 scope.go:122] "RemoveContainer" containerID="9911ececf9901cae8da804db45c617ec5ed6abf2e8a34317b6988827504dbed8" Oct 30 18:42:04.891846 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:42:04.891830 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9911ececf9901cae8da804db45c617ec5ed6abf2e8a34317b6988827504dbed8\": container with ID starting with 9911ececf9901cae8da804db45c617ec5ed6abf2e8a34317b6988827504dbed8 not found: ID does not exist" containerID="9911ececf9901cae8da804db45c617ec5ed6abf2e8a34317b6988827504dbed8" Oct 30 18:42:04.891903 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.891852 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9911ececf9901cae8da804db45c617ec5ed6abf2e8a34317b6988827504dbed8"} err="failed to get container status \"9911ececf9901cae8da804db45c617ec5ed6abf2e8a34317b6988827504dbed8\": rpc error: code = NotFound desc = could not find container \"9911ececf9901cae8da804db45c617ec5ed6abf2e8a34317b6988827504dbed8\": container with ID starting with 9911ececf9901cae8da804db45c617ec5ed6abf2e8a34317b6988827504dbed8 not found: ID does not exist" Oct 30 18:42:04.923679 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.923650 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj"] Oct 30 18:42:04.943547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:04.936544 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-f57fb7-openshift-default-5647675bf-kcflj"] Oct 30 18:42:05.023612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:05.023569 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h"] Oct 30 18:42:05.026138 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:05.026110 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-xhh7h"] Oct 30 18:42:06.800873 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:06.800842 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" path="/var/lib/kubelet/pods/4a5e3f2b-026e-4871-89ce-7ac9ed0f629c/volumes" Oct 30 18:42:06.801280 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:06.801244 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eaf5cf88-7d58-49a4-8dba-e56ede8eb25e" path="/var/lib/kubelet/pods/eaf5cf88-7d58-49a4-8dba-e56ede8eb25e/volumes" Oct 30 18:42:13.407130 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:13.407091 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" podUID="99e4007d-cd9b-4950-a80b-0383274c80f3" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:42:13.407130 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:13.407103 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" podUID="2cdf9ceb-4144-4694-a24f-b73ae0a06f96" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:42:14.927275 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:14.927240 5232 generic.go:356] "Generic (PLEG): container finished" podID="2cdf9ceb-4144-4694-a24f-b73ae0a06f96" containerID="4a3ff18848b8199ba42add282dd0d170399cea37b333c33bfe01475664234a91" exitCode=0 Oct 30 18:42:14.927705 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:14.927352 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" event={"ID":"2cdf9ceb-4144-4694-a24f-b73ae0a06f96","Type":"ContainerDied","Data":"4a3ff18848b8199ba42add282dd0d170399cea37b333c33bfe01475664234a91"} Oct 30 18:42:14.929449 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:14.929364 5232 generic.go:356] "Generic (PLEG): container finished" podID="99e4007d-cd9b-4950-a80b-0383274c80f3" containerID="2c3f1a8336394eb6d570670baff96eac11aaf2565f113c058c57afff03f25a25" exitCode=0 Oct 30 18:42:14.929449 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:14.929409 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" event={"ID":"99e4007d-cd9b-4950-a80b-0383274c80f3","Type":"ContainerDied","Data":"2c3f1a8336394eb6d570670baff96eac11aaf2565f113c058c57afff03f25a25"} Oct 30 18:42:15.035442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.035417 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" Oct 30 18:42:15.055442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.055417 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" Oct 30 18:42:15.194175 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.194140 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/99e4007d-cd9b-4950-a80b-0383274c80f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e4007d-cd9b-4950-a80b-0383274c80f3-plugins-config-volume\") pod \"99e4007d-cd9b-4950-a80b-0383274c80f3\" (UID: \"99e4007d-cd9b-4950-a80b-0383274c80f3\") " Oct 30 18:42:15.194391 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.194194 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2cdf9ceb-4144-4694-a24f-b73ae0a06f96-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cdf9ceb-4144-4694-a24f-b73ae0a06f96-plugins-config-volume\") pod \"2cdf9ceb-4144-4694-a24f-b73ae0a06f96\" (UID: \"2cdf9ceb-4144-4694-a24f-b73ae0a06f96\") " Oct 30 18:42:15.194391 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.194267 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2cdf9ceb-4144-4694-a24f-b73ae0a06f96-kube-api-access-5nqbd\" (UniqueName: \"kubernetes.io/projected/2cdf9ceb-4144-4694-a24f-b73ae0a06f96-kube-api-access-5nqbd\") pod \"2cdf9ceb-4144-4694-a24f-b73ae0a06f96\" (UID: \"2cdf9ceb-4144-4694-a24f-b73ae0a06f96\") " Oct 30 18:42:15.194391 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.194300 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/99e4007d-cd9b-4950-a80b-0383274c80f3-kube-api-access-m65x8\" (UniqueName: \"kubernetes.io/projected/99e4007d-cd9b-4950-a80b-0383274c80f3-kube-api-access-m65x8\") pod \"99e4007d-cd9b-4950-a80b-0383274c80f3\" (UID: \"99e4007d-cd9b-4950-a80b-0383274c80f3\") " Oct 30 18:42:15.194658 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.194630 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99e4007d-cd9b-4950-a80b-0383274c80f3-plugins-config-volume" pod "99e4007d-cd9b-4950-a80b-0383274c80f3" (UID: "99e4007d-cd9b-4950-a80b-0383274c80f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:15.194709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.194635 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2cdf9ceb-4144-4694-a24f-b73ae0a06f96-plugins-config-volume" pod "2cdf9ceb-4144-4694-a24f-b73ae0a06f96" (UID: "2cdf9ceb-4144-4694-a24f-b73ae0a06f96"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:42:15.196556 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.196534 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99e4007d-cd9b-4950-a80b-0383274c80f3-kube-api-access-m65x8" pod "99e4007d-cd9b-4950-a80b-0383274c80f3" (UID: "99e4007d-cd9b-4950-a80b-0383274c80f3"). InnerVolumeSpecName "kube-api-access-m65x8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:15.196751 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.196732 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2cdf9ceb-4144-4694-a24f-b73ae0a06f96-kube-api-access-5nqbd" pod "2cdf9ceb-4144-4694-a24f-b73ae0a06f96" (UID: "2cdf9ceb-4144-4694-a24f-b73ae0a06f96"). InnerVolumeSpecName "kube-api-access-5nqbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:42:15.295430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.295345 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/99e4007d-cd9b-4950-a80b-0383274c80f3-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:42:15.295430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.295378 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2cdf9ceb-4144-4694-a24f-b73ae0a06f96-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:42:15.295430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.295388 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5nqbd\" (UniqueName: \"kubernetes.io/projected/2cdf9ceb-4144-4694-a24f-b73ae0a06f96-kube-api-access-5nqbd\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:42:15.295430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.295398 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m65x8\" (UniqueName: \"kubernetes.io/projected/99e4007d-cd9b-4950-a80b-0383274c80f3-kube-api-access-m65x8\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:42:15.935142 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.935108 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" Oct 30 18:42:15.935661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.935144 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw" event={"ID":"99e4007d-cd9b-4950-a80b-0383274c80f3","Type":"ContainerDied","Data":"f54ccd16e50dfc40ee1ffd098da13b9a7b9a65cc65f0913de0b83ee54bb76b1f"} Oct 30 18:42:15.935661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.935217 5232 scope.go:122] "RemoveContainer" containerID="2c3f1a8336394eb6d570670baff96eac11aaf2565f113c058c57afff03f25a25" Oct 30 18:42:15.936798 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.936496 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" event={"ID":"2cdf9ceb-4144-4694-a24f-b73ae0a06f96","Type":"ContainerDied","Data":"c0bf24885c79e436e0d1ecf255c4d8218e349d560795377a5a580a395c57ef01"} Oct 30 18:42:15.936798 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.936572 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv" Oct 30 18:42:15.945317 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.945295 5232 scope.go:122] "RemoveContainer" containerID="4a3ff18848b8199ba42add282dd0d170399cea37b333c33bfe01475664234a91" Oct 30 18:42:15.984754 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.984729 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw"] Oct 30 18:42:15.996447 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:15.996422 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-f57fb7-75558bd4966hzw"] Oct 30 18:42:16.021489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:16.021427 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv"] Oct 30 18:42:16.027298 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:16.027274 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-f57fb7-6b584qd4tv"] Oct 30 18:42:16.033002 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:16.032980 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb"] Oct 30 18:42:16.035631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:16.035608 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-rrzmb"] Oct 30 18:42:16.800668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:16.800634 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2cdf9ceb-4144-4694-a24f-b73ae0a06f96" path="/var/lib/kubelet/pods/2cdf9ceb-4144-4694-a24f-b73ae0a06f96/volumes" Oct 30 18:42:16.801045 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:16.801032 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5fac5b90-e6ca-4a1f-9014-61149123cc5f" path="/var/lib/kubelet/pods/5fac5b90-e6ca-4a1f-9014-61149123cc5f/volumes" Oct 30 18:42:16.801350 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:16.801338 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="99e4007d-cd9b-4950-a80b-0383274c80f3" path="/var/lib/kubelet/pods/99e4007d-cd9b-4950-a80b-0383274c80f3/volumes" Oct 30 18:42:27.031765 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:42:27.031671 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:42:41.032384 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:41.032350 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5"] Oct 30 18:42:41.044292 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:41.044263 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-mmnn5"] Oct 30 18:42:42.800650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:42.800621 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fcdcc782-a4b3-455e-b95e-fe3d5011093f" path="/var/lib/kubelet/pods/fcdcc782-a4b3-455e-b95e-fe3d5011093f/volumes" Oct 30 18:42:55.689217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:55.689184 5232 scope.go:122] "RemoveContainer" containerID="af60d57f9c152e17172d6de362b6935d9a822a87fcf07b272368cb6951fdd5d4" Oct 30 18:42:55.698756 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:42:55.698727 5232 scope.go:122] "RemoveContainer" containerID="f4b25b24148dc653694ae1625be617c99f6b96725ca027aa14427ca1a3d05b2a" Oct 30 18:42:57.032689 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:42:57.032646 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:43:06.032243 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:06.032204 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp"] Oct 30 18:43:06.033583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:06.033559 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-7frnp"] Oct 30 18:43:06.800749 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:06.800715 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e13ef4dd-0358-469c-a655-d447f298aec2" path="/var/lib/kubelet/pods/e13ef4dd-0358-469c-a655-d447f298aec2/volumes" Oct 30 18:43:27.033025 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:43:27.032986 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:43:30.025162 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:30.025128 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n"] Oct 30 18:43:30.036962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:30.036930 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-r6w7n"] Oct 30 18:43:30.800612 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:30.800579 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ef6f72d-739e-4f95-83de-46f03c3dbaa3" path="/var/lib/kubelet/pods/6ef6f72d-739e-4f95-83de-46f03c3dbaa3/volumes" Oct 30 18:43:46.871600 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:46.871557 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc"] Oct 30 18:43:46.872440 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:46.872415 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7044133a-08fc-4861-b938-941ea0a59b2d" containerName="revoke-keys" Oct 30 18:43:46.872594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:46.872442 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7044133a-08fc-4861-b938-941ea0a59b2d" containerName="revoke-keys" Oct 30 18:43:46.872594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:46.872461 5232 cpu_manager.go:414] "containerMap: removing container" podUID="2cdf9ceb-4144-4694-a24f-b73ae0a06f96" containerName="payload-pre-processing" Oct 30 18:43:46.872594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:46.872468 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2cdf9ceb-4144-4694-a24f-b73ae0a06f96" containerName="payload-pre-processing" Oct 30 18:43:46.872594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:46.872487 5232 cpu_manager.go:414] "containerMap: removing container" podUID="99e4007d-cd9b-4950-a80b-0383274c80f3" containerName="payload-processing" Oct 30 18:43:46.872594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:46.872494 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="99e4007d-cd9b-4950-a80b-0383274c80f3" containerName="payload-processing" Oct 30 18:43:46.872594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:46.872537 5232 cpu_manager.go:414] "containerMap: removing container" podUID="4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" containerName="istio-proxy" Oct 30 18:43:46.872594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:46.872545 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" containerName="istio-proxy" Oct 30 18:43:46.872969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:46.872650 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7044133a-08fc-4861-b938-941ea0a59b2d" containerName="revoke-keys" Oct 30 18:43:46.872969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:46.872665 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="2cdf9ceb-4144-4694-a24f-b73ae0a06f96" containerName="payload-pre-processing" Oct 30 18:43:46.872969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:46.872675 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="99e4007d-cd9b-4950-a80b-0383274c80f3" containerName="payload-processing" Oct 30 18:43:46.872969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:46.872688 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" containerName="istio-proxy" Oct 30 18:43:46.872969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:46.872698 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" containerName="istio-proxy" Oct 30 18:43:46.875826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:46.875804 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc" Oct 30 18:43:46.892093 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:46.892062 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc"] Oct 30 18:43:46.902839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:46.902817 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:43:46.902969 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:46.902859 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:43:47.048921 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:47.048885 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffd8a90-7349-4064-9c07-a64fec195657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-nr4nc\" (UID: \"dffd8a90-7349-4064-9c07-a64fec195657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc" Oct 30 18:43:47.049092 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:47.048935 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c9n5p\" (UniqueName: \"kubernetes.io/projected/dffd8a90-7349-4064-9c07-a64fec195657-kube-api-access-c9n5p\") pod \"maas-api-revoke-sub-d576e68b-nr4nc\" (UID: \"dffd8a90-7349-4064-9c07-a64fec195657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc" Oct 30 18:43:47.049092 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:47.049037 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffd8a90-7349-4064-9c07-a64fec195657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-nr4nc\" (UID: \"dffd8a90-7349-4064-9c07-a64fec195657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc" Oct 30 18:43:47.150413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:47.150319 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffd8a90-7349-4064-9c07-a64fec195657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-nr4nc\" (UID: \"dffd8a90-7349-4064-9c07-a64fec195657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc" Oct 30 18:43:47.150413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:47.150382 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c9n5p\" (UniqueName: \"kubernetes.io/projected/dffd8a90-7349-4064-9c07-a64fec195657-kube-api-access-c9n5p\") pod \"maas-api-revoke-sub-d576e68b-nr4nc\" (UID: \"dffd8a90-7349-4064-9c07-a64fec195657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc" Oct 30 18:43:47.150413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:47.150414 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffd8a90-7349-4064-9c07-a64fec195657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-nr4nc\" (UID: \"dffd8a90-7349-4064-9c07-a64fec195657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc" Oct 30 18:43:47.151047 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:47.151027 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffd8a90-7349-4064-9c07-a64fec195657-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-nr4nc\" (UID: \"dffd8a90-7349-4064-9c07-a64fec195657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc" Oct 30 18:43:47.152812 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:47.152786 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffd8a90-7349-4064-9c07-a64fec195657-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-nr4nc\" (UID: \"dffd8a90-7349-4064-9c07-a64fec195657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc" Oct 30 18:43:47.169962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:47.169937 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c9n5p\" (UniqueName: \"kubernetes.io/projected/dffd8a90-7349-4064-9c07-a64fec195657-kube-api-access-c9n5p\") pod \"maas-api-revoke-sub-d576e68b-nr4nc\" (UID: \"dffd8a90-7349-4064-9c07-a64fec195657\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc" Oct 30 18:43:47.187754 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:47.187731 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc" Oct 30 18:43:47.476629 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:47.476604 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc"] Oct 30 18:43:47.478432 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:43:47.478399 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddffd8a90_7349_4064_9c07_a64fec195657.slice/crio-8edc64a0d1967832eb94b04d91239a422972e53dbef9aea9ddff7db2bcf64674 WatchSource:0}: Error finding container 8edc64a0d1967832eb94b04d91239a422972e53dbef9aea9ddff7db2bcf64674: Status 404 returned error can't find the container with id 8edc64a0d1967832eb94b04d91239a422972e53dbef9aea9ddff7db2bcf64674 Oct 30 18:43:48.380905 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:48.380873 5232 generic.go:356] "Generic (PLEG): container finished" podID="dffd8a90-7349-4064-9c07-a64fec195657" containerID="900e137bd29fc72adc637b37725cfaaab4056c9bf91c835b86d411fbfcd895b7" exitCode=0 Oct 30 18:43:48.381299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:48.380958 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc" event={"ID":"dffd8a90-7349-4064-9c07-a64fec195657","Type":"ContainerDied","Data":"900e137bd29fc72adc637b37725cfaaab4056c9bf91c835b86d411fbfcd895b7"} Oct 30 18:43:48.381299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:48.381042 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc" event={"ID":"dffd8a90-7349-4064-9c07-a64fec195657","Type":"ContainerStarted","Data":"8edc64a0d1967832eb94b04d91239a422972e53dbef9aea9ddff7db2bcf64674"} Oct 30 18:43:49.514314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:49.514286 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc" Oct 30 18:43:49.675783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:49.675706 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dffd8a90-7349-4064-9c07-a64fec195657-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffd8a90-7349-4064-9c07-a64fec195657-maas-api-openshift-ca\") pod \"dffd8a90-7349-4064-9c07-a64fec195657\" (UID: \"dffd8a90-7349-4064-9c07-a64fec195657\") " Oct 30 18:43:49.675783 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:49.675775 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dffd8a90-7349-4064-9c07-a64fec195657-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffd8a90-7349-4064-9c07-a64fec195657-maas-api-xks-ca\") pod \"dffd8a90-7349-4064-9c07-a64fec195657\" (UID: \"dffd8a90-7349-4064-9c07-a64fec195657\") " Oct 30 18:43:49.675978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:49.675836 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dffd8a90-7349-4064-9c07-a64fec195657-kube-api-access-c9n5p\" (UniqueName: \"kubernetes.io/projected/dffd8a90-7349-4064-9c07-a64fec195657-kube-api-access-c9n5p\") pod \"dffd8a90-7349-4064-9c07-a64fec195657\" (UID: \"dffd8a90-7349-4064-9c07-a64fec195657\") " Oct 30 18:43:49.676100 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:49.676075 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dffd8a90-7349-4064-9c07-a64fec195657-maas-api-openshift-ca" pod "dffd8a90-7349-4064-9c07-a64fec195657" (UID: "dffd8a90-7349-4064-9c07-a64fec195657"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:43:49.677974 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:49.677946 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dffd8a90-7349-4064-9c07-a64fec195657-maas-api-xks-ca" pod "dffd8a90-7349-4064-9c07-a64fec195657" (UID: "dffd8a90-7349-4064-9c07-a64fec195657"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:43:49.678095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:49.678010 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dffd8a90-7349-4064-9c07-a64fec195657-kube-api-access-c9n5p" pod "dffd8a90-7349-4064-9c07-a64fec195657" (UID: "dffd8a90-7349-4064-9c07-a64fec195657"). InnerVolumeSpecName "kube-api-access-c9n5p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:43:49.776750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:49.776713 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c9n5p\" (UniqueName: \"kubernetes.io/projected/dffd8a90-7349-4064-9c07-a64fec195657-kube-api-access-c9n5p\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:43:49.776750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:49.776744 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dffd8a90-7349-4064-9c07-a64fec195657-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:43:49.776750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:49.776754 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dffd8a90-7349-4064-9c07-a64fec195657-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:43:50.025012 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:50.024976 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6"] Oct 30 18:43:50.037287 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:50.037260 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-rbdq6"] Oct 30 18:43:50.391476 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:50.391403 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc" Oct 30 18:43:50.391476 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:50.391416 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc" event={"ID":"dffd8a90-7349-4064-9c07-a64fec195657","Type":"ContainerDied","Data":"8edc64a0d1967832eb94b04d91239a422972e53dbef9aea9ddff7db2bcf64674"} Oct 30 18:43:50.391476 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:50.391458 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8edc64a0d1967832eb94b04d91239a422972e53dbef9aea9ddff7db2bcf64674" Oct 30 18:43:50.800767 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:50.800737 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aacc7754-1a56-4c14-9e31-2afad4653145" path="/var/lib/kubelet/pods/aacc7754-1a56-4c14-9e31-2afad4653145/volumes" Oct 30 18:43:55.735838 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:55.735810 5232 scope.go:122] "RemoveContainer" containerID="cde04c97451bacc0bbc6aeb071c3d41e723953e54a6da7a0452a18791bf23110" Oct 30 18:43:55.744272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:55.744254 5232 scope.go:122] "RemoveContainer" containerID="89d76d33557966a5d8352082279b5ccfb72f74fceeb3a67256155b227f92cf99" Oct 30 18:43:55.751942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:43:55.751924 5232 scope.go:122] "RemoveContainer" containerID="dcb3cf9a20af9a2f28f7afd08377e1924e24309ce48c662fad8137d195a34002" Oct 30 18:43:57.033275 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:43:57.033231 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:44:11.440205 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.440172 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch"] Oct 30 18:44:11.440620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.440552 5232 cpu_manager.go:414] "containerMap: removing container" podUID="dffd8a90-7349-4064-9c07-a64fec195657" containerName="revoke-keys" Oct 30 18:44:11.440620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.440566 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dffd8a90-7349-4064-9c07-a64fec195657" containerName="revoke-keys" Oct 30 18:44:11.440620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.440595 5232 cpu_manager.go:414] "containerMap: removing container" podUID="4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" containerName="istio-proxy" Oct 30 18:44:11.440620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.440600 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4a5e3f2b-026e-4871-89ce-7ac9ed0f629c" containerName="istio-proxy" Oct 30 18:44:11.440781 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.440664 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="dffd8a90-7349-4064-9c07-a64fec195657" containerName="revoke-keys" Oct 30 18:44:11.443661 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.443646 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch" Oct 30 18:44:11.461445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.461418 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch"] Oct 30 18:44:11.471850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.471809 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:44:11.471993 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.471862 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:44:11.490106 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.490081 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd60005-71c6-41cf-9c41-e4dafe67490c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-z9gch\" (UID: \"1cd60005-71c6-41cf-9c41-e4dafe67490c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch" Oct 30 18:44:11.490215 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.490113 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd60005-71c6-41cf-9c41-e4dafe67490c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-z9gch\" (UID: \"1cd60005-71c6-41cf-9c41-e4dafe67490c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch" Oct 30 18:44:11.490215 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.490155 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7r9bq\" (UniqueName: \"kubernetes.io/projected/1cd60005-71c6-41cf-9c41-e4dafe67490c-kube-api-access-7r9bq\") pod \"maas-api-revoke-sub-d2bc93bc-z9gch\" (UID: \"1cd60005-71c6-41cf-9c41-e4dafe67490c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch" Oct 30 18:44:11.590746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.590709 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd60005-71c6-41cf-9c41-e4dafe67490c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-z9gch\" (UID: \"1cd60005-71c6-41cf-9c41-e4dafe67490c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch" Oct 30 18:44:11.590947 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.590756 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd60005-71c6-41cf-9c41-e4dafe67490c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-z9gch\" (UID: \"1cd60005-71c6-41cf-9c41-e4dafe67490c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch" Oct 30 18:44:11.590947 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.590815 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7r9bq\" (UniqueName: \"kubernetes.io/projected/1cd60005-71c6-41cf-9c41-e4dafe67490c-kube-api-access-7r9bq\") pod \"maas-api-revoke-sub-d2bc93bc-z9gch\" (UID: \"1cd60005-71c6-41cf-9c41-e4dafe67490c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch" Oct 30 18:44:11.591446 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.591425 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd60005-71c6-41cf-9c41-e4dafe67490c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-z9gch\" (UID: \"1cd60005-71c6-41cf-9c41-e4dafe67490c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch" Oct 30 18:44:11.593213 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.593194 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd60005-71c6-41cf-9c41-e4dafe67490c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-z9gch\" (UID: \"1cd60005-71c6-41cf-9c41-e4dafe67490c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch" Oct 30 18:44:11.610368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.610343 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7r9bq\" (UniqueName: \"kubernetes.io/projected/1cd60005-71c6-41cf-9c41-e4dafe67490c-kube-api-access-7r9bq\") pod \"maas-api-revoke-sub-d2bc93bc-z9gch\" (UID: \"1cd60005-71c6-41cf-9c41-e4dafe67490c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch" Oct 30 18:44:11.755930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:11.755896 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch" Oct 30 18:44:12.052692 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:12.052666 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch"] Oct 30 18:44:12.054603 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:44:12.054556 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1cd60005_71c6_41cf_9c41_e4dafe67490c.slice/crio-d990b9ff32f18b00c35f03cdf626653cc6fd3464b75696805c06db7e57433b90 WatchSource:0}: Error finding container d990b9ff32f18b00c35f03cdf626653cc6fd3464b75696805c06db7e57433b90: Status 404 returned error can't find the container with id d990b9ff32f18b00c35f03cdf626653cc6fd3464b75696805c06db7e57433b90 Oct 30 18:44:12.513786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:12.513749 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch" event={"ID":"1cd60005-71c6-41cf-9c41-e4dafe67490c","Type":"ContainerStarted","Data":"d923b817d58ace855bb9327ae9f361ab00fab0057936482958135981bc80dcac"} Oct 30 18:44:12.514135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:12.513815 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch" event={"ID":"1cd60005-71c6-41cf-9c41-e4dafe67490c","Type":"ContainerStarted","Data":"d990b9ff32f18b00c35f03cdf626653cc6fd3464b75696805c06db7e57433b90"} Oct 30 18:44:12.551700 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:12.551652 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch" podStartSLOduration=1.5516387680000001 podStartE2EDuration="1.551638768s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:12.550583296 +0000 UTC m=+2551.867003617" watchObservedRunningTime="2026-10-30 18:44:12.551638768 +0000 UTC m=+2551.868059083" Oct 30 18:44:13.520107 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:13.520073 5232 generic.go:356] "Generic (PLEG): container finished" podID="1cd60005-71c6-41cf-9c41-e4dafe67490c" containerID="d923b817d58ace855bb9327ae9f361ab00fab0057936482958135981bc80dcac" exitCode=0 Oct 30 18:44:13.520613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:13.520159 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch" event={"ID":"1cd60005-71c6-41cf-9c41-e4dafe67490c","Type":"ContainerDied","Data":"d923b817d58ace855bb9327ae9f361ab00fab0057936482958135981bc80dcac"} Oct 30 18:44:14.659072 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:14.659041 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch" Oct 30 18:44:14.718389 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:14.718355 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1cd60005-71c6-41cf-9c41-e4dafe67490c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd60005-71c6-41cf-9c41-e4dafe67490c-maas-api-xks-ca\") pod \"1cd60005-71c6-41cf-9c41-e4dafe67490c\" (UID: \"1cd60005-71c6-41cf-9c41-e4dafe67490c\") " Oct 30 18:44:14.718547 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:14.718529 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1cd60005-71c6-41cf-9c41-e4dafe67490c-kube-api-access-7r9bq\" (UniqueName: \"kubernetes.io/projected/1cd60005-71c6-41cf-9c41-e4dafe67490c-kube-api-access-7r9bq\") pod \"1cd60005-71c6-41cf-9c41-e4dafe67490c\" (UID: \"1cd60005-71c6-41cf-9c41-e4dafe67490c\") " Oct 30 18:44:14.718596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:14.718576 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1cd60005-71c6-41cf-9c41-e4dafe67490c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd60005-71c6-41cf-9c41-e4dafe67490c-maas-api-openshift-ca\") pod \"1cd60005-71c6-41cf-9c41-e4dafe67490c\" (UID: \"1cd60005-71c6-41cf-9c41-e4dafe67490c\") " Oct 30 18:44:14.718960 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:14.718936 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1cd60005-71c6-41cf-9c41-e4dafe67490c-maas-api-openshift-ca" pod "1cd60005-71c6-41cf-9c41-e4dafe67490c" (UID: "1cd60005-71c6-41cf-9c41-e4dafe67490c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:44:14.720564 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:14.720541 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1cd60005-71c6-41cf-9c41-e4dafe67490c-maas-api-xks-ca" pod "1cd60005-71c6-41cf-9c41-e4dafe67490c" (UID: "1cd60005-71c6-41cf-9c41-e4dafe67490c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:44:14.720722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:14.720707 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1cd60005-71c6-41cf-9c41-e4dafe67490c-kube-api-access-7r9bq" pod "1cd60005-71c6-41cf-9c41-e4dafe67490c" (UID: "1cd60005-71c6-41cf-9c41-e4dafe67490c"). InnerVolumeSpecName "kube-api-access-7r9bq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:44:14.820198 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:14.820169 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7r9bq\" (UniqueName: \"kubernetes.io/projected/1cd60005-71c6-41cf-9c41-e4dafe67490c-kube-api-access-7r9bq\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:44:14.820198 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:14.820196 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1cd60005-71c6-41cf-9c41-e4dafe67490c-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:44:14.820360 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:14.820205 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1cd60005-71c6-41cf-9c41-e4dafe67490c-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:44:15.530966 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:15.530934 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch" Oct 30 18:44:15.531239 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:15.530965 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch" event={"ID":"1cd60005-71c6-41cf-9c41-e4dafe67490c","Type":"ContainerDied","Data":"d990b9ff32f18b00c35f03cdf626653cc6fd3464b75696805c06db7e57433b90"} Oct 30 18:44:15.531239 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:15.531014 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d990b9ff32f18b00c35f03cdf626653cc6fd3464b75696805c06db7e57433b90" Oct 30 18:44:23.781792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:23.781759 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs"] Oct 30 18:44:23.782213 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:23.782109 5232 cpu_manager.go:414] "containerMap: removing container" podUID="1cd60005-71c6-41cf-9c41-e4dafe67490c" containerName="revoke-keys" Oct 30 18:44:23.782213 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:23.782119 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1cd60005-71c6-41cf-9c41-e4dafe67490c" containerName="revoke-keys" Oct 30 18:44:23.782213 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:23.782182 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="1cd60005-71c6-41cf-9c41-e4dafe67490c" containerName="revoke-keys" Oct 30 18:44:23.785720 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:23.785695 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:23.812304 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:23.812276 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs"] Oct 30 18:44:23.862476 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:23.862447 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-ddcab1-openshift-default-dockercfg-jtzz4\"" Oct 30 18:44:23.898623 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:23.898573 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-data\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:23.898850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:23.898659 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-workload-socket\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:23.898850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:23.898776 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-podinfo\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:23.898850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:23.898805 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-credential-socket\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:23.898850 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:23.898832 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-workload-certs\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:23.899098 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:23.898902 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-envoy\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:23.899098 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:23.898957 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tm2ln\" (UniqueName: \"kubernetes.io/projected/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-kube-api-access-tm2ln\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:23.899098 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:23.899006 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-token\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:23.899262 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:23.899147 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istiod-ca-cert\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.000575 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.000533 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-data\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.000786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.000583 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-workload-socket\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.000786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.000651 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-podinfo\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.000786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.000676 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-credential-socket\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.000786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.000701 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-workload-certs\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.000786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.000756 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-envoy\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.000786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.000779 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tm2ln\" (UniqueName: \"kubernetes.io/projected/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-kube-api-access-tm2ln\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.001122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.000816 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-token\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.001122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.000850 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istiod-ca-cert\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.001122 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.000997 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-data\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.001295 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.001200 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-workload-socket\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.001467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.001440 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-credential-socket\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.001650 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.001627 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istiod-ca-cert\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.001725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.001682 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-workload-certs\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.003341 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.003320 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-envoy\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.003341 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.003335 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-podinfo\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.020055 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.020024 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-token\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.020193 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.020172 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tm2ln\" (UniqueName: \"kubernetes.io/projected/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-kube-api-access-tm2ln\") pod \"e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.098709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.098623 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:24.400435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.400372 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs"] Oct 30 18:44:24.402801 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:44:24.402773 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ec1dda5_2fcf_444a_b6c2_40ac0e64ba25.slice/crio-3cb58c0857e15299d3246736d15490384c4d21072564c04e100aa3f1af77b2a1 WatchSource:0}: Error finding container 3cb58c0857e15299d3246736d15490384c4d21072564c04e100aa3f1af77b2a1: Status 404 returned error can't find the container with id 3cb58c0857e15299d3246736d15490384c4d21072564c04e100aa3f1af77b2a1 Oct 30 18:44:24.404973 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.404935 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:44:24.405103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.405017 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:44:24.405103 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.405061 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:44:24.577111 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.577071 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" event={"ID":"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25","Type":"ContainerStarted","Data":"0fbaa748638e0361cff8ded0802098f8221691f98d304a33a288c267d4c1e321"} Oct 30 18:44:24.577265 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.577123 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" event={"ID":"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25","Type":"ContainerStarted","Data":"3cb58c0857e15299d3246736d15490384c4d21072564c04e100aa3f1af77b2a1"} Oct 30 18:44:24.620053 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:24.620007 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" podStartSLOduration=1.619993238 podStartE2EDuration="1.619993238s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:24.617659881 +0000 UTC m=+2563.934080277" watchObservedRunningTime="2026-10-30 18:44:24.619993238 +0000 UTC m=+2563.936413550" Oct 30 18:44:25.099204 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:25.099168 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:25.104408 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:25.104385 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:25.581406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:25.581374 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:25.582330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:25.582312 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:27.032417 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:44:27.032379 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:44:28.435995 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.435964 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln"] Oct 30 18:44:28.439492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.439466 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" Oct 30 18:44:28.468256 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.468225 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ddcab1-serving-cert\"" Oct 30 18:44:28.468450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.468281 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:44:28.468660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.468624 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpqx\"" Oct 30 18:44:28.470195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.470174 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln"] Oct 30 18:44:28.545148 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.545114 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ef8aad-dce5-41ca-826d-7fedab6368b4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln\" (UID: \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" Oct 30 18:44:28.545321 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.545168 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksj8l\" (UniqueName: \"kubernetes.io/projected/d5ef8aad-dce5-41ca-826d-7fedab6368b4-kube-api-access-ksj8l\") pod \"maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln\" (UID: \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" Oct 30 18:44:28.545321 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.545303 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ef8aad-dce5-41ca-826d-7fedab6368b4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln\" (UID: \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" Oct 30 18:44:28.545399 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.545353 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ef8aad-dce5-41ca-826d-7fedab6368b4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln\" (UID: \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" Oct 30 18:44:28.646631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.646593 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ef8aad-dce5-41ca-826d-7fedab6368b4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln\" (UID: \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" Oct 30 18:44:28.646821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.646660 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksj8l\" (UniqueName: \"kubernetes.io/projected/d5ef8aad-dce5-41ca-826d-7fedab6368b4-kube-api-access-ksj8l\") pod \"maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln\" (UID: \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" Oct 30 18:44:28.646821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.646743 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ef8aad-dce5-41ca-826d-7fedab6368b4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln\" (UID: \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" Oct 30 18:44:28.646821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.646796 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ef8aad-dce5-41ca-826d-7fedab6368b4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln\" (UID: \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" Oct 30 18:44:28.647444 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.647415 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ef8aad-dce5-41ca-826d-7fedab6368b4-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln\" (UID: \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" Oct 30 18:44:28.650095 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.650070 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ef8aad-dce5-41ca-826d-7fedab6368b4-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln\" (UID: \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" Oct 30 18:44:28.652212 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.652189 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ef8aad-dce5-41ca-826d-7fedab6368b4-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln\" (UID: \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" Oct 30 18:44:28.666097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.666072 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksj8l\" (UniqueName: \"kubernetes.io/projected/d5ef8aad-dce5-41ca-826d-7fedab6368b4-kube-api-access-ksj8l\") pod \"maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln\" (UID: \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" Oct 30 18:44:28.753157 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.753127 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" Oct 30 18:44:28.967994 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.967956 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2"] Oct 30 18:44:28.972425 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.972406 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" Oct 30 18:44:28.989003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.988976 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2"] Oct 30 18:44:28.990308 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.990287 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-ddcab1\"" Oct 30 18:44:28.990485 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:28.990436 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-388c85d0-dockercfg-g8krg\"" Oct 30 18:44:29.064606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.050817 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d1f62d-61a9-45b0-84f9-cb182481db06-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2\" (UID: \"61d1f62d-61a9-45b0-84f9-cb182481db06\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" Oct 30 18:44:29.064606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.050959 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9vls\" (UniqueName: \"kubernetes.io/projected/61d1f62d-61a9-45b0-84f9-cb182481db06-kube-api-access-g9vls\") pod \"payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2\" (UID: \"61d1f62d-61a9-45b0-84f9-cb182481db06\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" Oct 30 18:44:29.067686 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.067596 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln"] Oct 30 18:44:29.069832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.069788 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph"] Oct 30 18:44:29.073933 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.073908 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" Oct 30 18:44:29.077758 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:44:29.077728 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ef8aad_dce5_41ca_826d_7fedab6368b4.slice/crio-c486001ac2945cbae81c891023a1fe853617eb2a6d7e77ac4a7c8127fe749945 WatchSource:0}: Error finding container c486001ac2945cbae81c891023a1fe853617eb2a6d7e77ac4a7c8127fe749945: Status 404 returned error can't find the container with id c486001ac2945cbae81c891023a1fe853617eb2a6d7e77ac4a7c8127fe749945 Oct 30 18:44:29.092300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.092261 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph"] Oct 30 18:44:29.151980 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.151941 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71533836-7a75-4c67-a826-4106480146e5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph\" (UID: \"71533836-7a75-4c67-a826-4106480146e5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" Oct 30 18:44:29.152143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.152001 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dgbl\" (UniqueName: \"kubernetes.io/projected/71533836-7a75-4c67-a826-4106480146e5-kube-api-access-4dgbl\") pod \"payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph\" (UID: \"71533836-7a75-4c67-a826-4106480146e5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" Oct 30 18:44:29.152143 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.152117 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9vls\" (UniqueName: \"kubernetes.io/projected/61d1f62d-61a9-45b0-84f9-cb182481db06-kube-api-access-g9vls\") pod \"payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2\" (UID: \"61d1f62d-61a9-45b0-84f9-cb182481db06\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" Oct 30 18:44:29.152271 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.152223 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d1f62d-61a9-45b0-84f9-cb182481db06-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2\" (UID: \"61d1f62d-61a9-45b0-84f9-cb182481db06\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" Oct 30 18:44:29.152992 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.152959 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d1f62d-61a9-45b0-84f9-cb182481db06-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2\" (UID: \"61d1f62d-61a9-45b0-84f9-cb182481db06\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" Oct 30 18:44:29.171550 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.171499 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9vls\" (UniqueName: \"kubernetes.io/projected/61d1f62d-61a9-45b0-84f9-cb182481db06-kube-api-access-g9vls\") pod \"payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2\" (UID: \"61d1f62d-61a9-45b0-84f9-cb182481db06\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" Oct 30 18:44:29.253282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.253250 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71533836-7a75-4c67-a826-4106480146e5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph\" (UID: \"71533836-7a75-4c67-a826-4106480146e5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" Oct 30 18:44:29.253282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.253289 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dgbl\" (UniqueName: \"kubernetes.io/projected/71533836-7a75-4c67-a826-4106480146e5-kube-api-access-4dgbl\") pod \"payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph\" (UID: \"71533836-7a75-4c67-a826-4106480146e5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" Oct 30 18:44:29.254011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.253985 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71533836-7a75-4c67-a826-4106480146e5-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph\" (UID: \"71533836-7a75-4c67-a826-4106480146e5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" Oct 30 18:44:29.273828 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.273798 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dgbl\" (UniqueName: \"kubernetes.io/projected/71533836-7a75-4c67-a826-4106480146e5-kube-api-access-4dgbl\") pod \"payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph\" (UID: \"71533836-7a75-4c67-a826-4106480146e5\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" Oct 30 18:44:29.285424 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.285394 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" Oct 30 18:44:29.388116 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.388088 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" Oct 30 18:44:29.578314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.578290 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2"] Oct 30 18:44:29.579855 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:44:29.579827 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d1f62d_61a9_45b0_84f9_cb182481db06.slice/crio-f39059c296c8dbe103c376af34f6708b25882784a2dc2e9e67d7f9f3f10f653b WatchSource:0}: Error finding container f39059c296c8dbe103c376af34f6708b25882784a2dc2e9e67d7f9f3f10f653b: Status 404 returned error can't find the container with id f39059c296c8dbe103c376af34f6708b25882784a2dc2e9e67d7f9f3f10f653b Oct 30 18:44:29.601298 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.601271 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" event={"ID":"61d1f62d-61a9-45b0-84f9-cb182481db06","Type":"ContainerStarted","Data":"f39059c296c8dbe103c376af34f6708b25882784a2dc2e9e67d7f9f3f10f653b"} Oct 30 18:44:29.602733 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.602711 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" event={"ID":"d5ef8aad-dce5-41ca-826d-7fedab6368b4","Type":"ContainerStarted","Data":"7bd4c3af7262e49fa914e9e1cee11eaf6bff2802a7438d41560dcd3427899999"} Oct 30 18:44:29.602832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.602753 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" event={"ID":"d5ef8aad-dce5-41ca-826d-7fedab6368b4","Type":"ContainerStarted","Data":"c486001ac2945cbae81c891023a1fe853617eb2a6d7e77ac4a7c8127fe749945"} Oct 30 18:44:29.602880 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.602837 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" Oct 30 18:44:29.644637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.644596 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" podStartSLOduration=1.294388324 podStartE2EDuration="1.644582984s" totalImagesPullingTime="350.19466ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:29.643909228 +0000 UTC m=+2568.960329549" watchObservedRunningTime="2026-10-30 18:44:29.644582984 +0000 UTC m=+2568.961003350" Oct 30 18:44:29.681142 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:29.681079 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph"] Oct 30 18:44:29.683776 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:44:29.683742 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71533836_7a75_4c67_a826_4106480146e5.slice/crio-54a6186da6ab7ede1099613de207b067f2a87d3e30f0aab6b5714821bf899669 WatchSource:0}: Error finding container 54a6186da6ab7ede1099613de207b067f2a87d3e30f0aab6b5714821bf899669: Status 404 returned error can't find the container with id 54a6186da6ab7ede1099613de207b067f2a87d3e30f0aab6b5714821bf899669 Oct 30 18:44:30.609209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:30.609152 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" event={"ID":"61d1f62d-61a9-45b0-84f9-cb182481db06","Type":"ContainerStarted","Data":"7bd46686e4321e333df4728e9884fa036a62bcb9cbd511632c07a2e6cc2c1ee3"} Oct 30 18:44:30.609707 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:30.609226 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" Oct 30 18:44:30.610729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:30.610694 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" event={"ID":"71533836-7a75-4c67-a826-4106480146e5","Type":"ContainerStarted","Data":"a0f91980d800797315281eb4bc363cce5f6fbaa60ed05028b09ed6aa2686f28a"} Oct 30 18:44:30.610866 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:30.610752 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" event={"ID":"71533836-7a75-4c67-a826-4106480146e5","Type":"ContainerStarted","Data":"54a6186da6ab7ede1099613de207b067f2a87d3e30f0aab6b5714821bf899669"} Oct 30 18:44:30.610866 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:30.610803 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" Oct 30 18:44:30.646885 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:30.646841 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" podStartSLOduration=2.646828711 podStartE2EDuration="2.646828711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:30.646428466 +0000 UTC m=+2569.962848787" watchObservedRunningTime="2026-10-30 18:44:30.646828711 +0000 UTC m=+2569.963249032" Oct 30 18:44:30.684114 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:30.684074 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" podStartSLOduration=1.684063139 podStartE2EDuration="1.684063139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:30.683850784 +0000 UTC m=+2570.000271106" watchObservedRunningTime="2026-10-30 18:44:30.684063139 +0000 UTC m=+2570.000483483" Oct 30 18:44:35.635570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:35.635531 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" Oct 30 18:44:40.583210 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:40.583172 5232 patch_prober.go:29] interesting pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Oct 30 18:44:40.583712 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:40.583226 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" podUID="8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Oct 30 18:44:40.682337 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:40.682305 5232 generic.go:356] "Generic (PLEG): container finished" podID="8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" containerID="0fbaa748638e0361cff8ded0802098f8221691f98d304a33a288c267d4c1e321" exitCode=0 Oct 30 18:44:40.682571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:40.682384 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" event={"ID":"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25","Type":"ContainerDied","Data":"0fbaa748638e0361cff8ded0802098f8221691f98d304a33a288c267d4c1e321"} Oct 30 18:44:40.682861 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:40.682844 5232 scope.go:122] "RemoveContainer" containerID="0fbaa748638e0361cff8ded0802098f8221691f98d304a33a288c267d4c1e321" Oct 30 18:44:40.684112 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:40.684085 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:44:40.684216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:40.684172 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:44:40.684278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:40.684221 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:44:41.640596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:41.640568 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" Oct 30 18:44:41.641031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:41.640634 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" Oct 30 18:44:41.690923 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:41.690888 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" event={"ID":"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25","Type":"ContainerStarted","Data":"340795e28946235c90b66e155f2a4f9e6197738699428ca00c9d85ce3023204c"} Oct 30 18:44:42.099079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:42.099046 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:42.099282 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:42.099250 5232 patch_prober.go:29] interesting pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Oct 30 18:44:42.099330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:42.099297 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" podUID="8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Oct 30 18:44:43.099065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:43.099029 5232 patch_prober.go:29] interesting pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Oct 30 18:44:43.099472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:43.099102 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" podUID="8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Oct 30 18:44:43.833045 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:43.833008 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr"] Oct 30 18:44:43.837261 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:43.837235 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:43.864656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:43.864632 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ddcab1\"/\"kube-root-ca.crt\"" Oct 30 18:44:43.864851 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:43.864825 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-single-main-ddcab1\"/\"openshift-service-ca.crt\"" Oct 30 18:44:43.866806 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:43.866775 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr"] Oct 30 18:44:43.926051 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:43.926010 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-dshm\") pod \"simulator-single-main-ddcab1-kserve-668fc5466c-695xr\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:43.926499 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:43.926473 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-home\") pod \"simulator-single-main-ddcab1-kserve-668fc5466c-695xr\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:43.926729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:43.926712 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-tls-certs\") pod \"simulator-single-main-ddcab1-kserve-668fc5466c-695xr\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:43.930662 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:43.930633 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-model-cache\") pod \"simulator-single-main-ddcab1-kserve-668fc5466c-695xr\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:43.930915 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:43.930893 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmf5m\" (UniqueName: \"kubernetes.io/projected/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-kube-api-access-lmf5m\") pod \"simulator-single-main-ddcab1-kserve-668fc5466c-695xr\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:43.948712 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:43.948672 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-tmp-dir\") pod \"simulator-single-main-ddcab1-kserve-668fc5466c-695xr\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:43.949309 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:43.949285 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ddcab1\"/\"default-dockercfg-vtjr2\"" Oct 30 18:44:43.949462 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:43.949364 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-single-main-ddcab1\"/\"simulator-single-main-ddcab1-kserve-self-signed-certs\"" Oct 30 18:44:44.050429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.050395 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-home\") pod \"simulator-single-main-ddcab1-kserve-668fc5466c-695xr\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:44.050429 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.050430 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-tls-certs\") pod \"simulator-single-main-ddcab1-kserve-668fc5466c-695xr\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:44.050737 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.050454 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-model-cache\") pod \"simulator-single-main-ddcab1-kserve-668fc5466c-695xr\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:44.050737 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.050496 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmf5m\" (UniqueName: \"kubernetes.io/projected/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-kube-api-access-lmf5m\") pod \"simulator-single-main-ddcab1-kserve-668fc5466c-695xr\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:44.050737 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.050554 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-tmp-dir\") pod \"simulator-single-main-ddcab1-kserve-668fc5466c-695xr\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:44.050737 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.050623 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-dshm\") pod \"simulator-single-main-ddcab1-kserve-668fc5466c-695xr\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:44.050990 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.050866 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-home\") pod \"simulator-single-main-ddcab1-kserve-668fc5466c-695xr\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:44.050990 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.050911 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-model-cache\") pod \"simulator-single-main-ddcab1-kserve-668fc5466c-695xr\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:44.050990 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.050971 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-tmp-dir\") pod \"simulator-single-main-ddcab1-kserve-668fc5466c-695xr\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:44.053015 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.052991 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-dshm\") pod \"simulator-single-main-ddcab1-kserve-668fc5466c-695xr\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:44.053245 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.053231 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-tls-certs\") pod \"simulator-single-main-ddcab1-kserve-668fc5466c-695xr\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:44.069734 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.069664 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmf5m\" (UniqueName: \"kubernetes.io/projected/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-kube-api-access-lmf5m\") pod \"simulator-single-main-ddcab1-kserve-668fc5466c-695xr\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:44.099599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.099469 5232 patch_prober.go:29] interesting pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Oct 30 18:44:44.099599 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.099565 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" podUID="8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Oct 30 18:44:44.149869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.149830 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:44.449821 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.449795 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr"] Oct 30 18:44:44.451888 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:44:44.451857 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod261c1e8a_47e6_4ef3_8c59_fe509cc5d315.slice/crio-9f002980aa477dd6d2eb91ae85550de466718346c57703c065813846d34c6895 WatchSource:0}: Error finding container 9f002980aa477dd6d2eb91ae85550de466718346c57703c065813846d34c6895: Status 404 returned error can't find the container with id 9f002980aa477dd6d2eb91ae85550de466718346c57703c065813846d34c6895 Oct 30 18:44:44.708631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.708534 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" event={"ID":"261c1e8a-47e6-4ef3-8c59-fe509cc5d315","Type":"ContainerStarted","Data":"96528e8913832022e8bbc626af7afd8d41fa0d28f0bde233db3150bbe8671420"} Oct 30 18:44:44.708631 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.708586 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" event={"ID":"261c1e8a-47e6-4ef3-8c59-fe509cc5d315","Type":"ContainerStarted","Data":"9f002980aa477dd6d2eb91ae85550de466718346c57703c065813846d34c6895"} Oct 30 18:44:44.758411 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:44.758351 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" podStartSLOduration=1.758334981 podStartE2EDuration="1.758334981s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:44:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:44:44.757639919 +0000 UTC m=+2584.074060242" watchObservedRunningTime="2026-10-30 18:44:44.758334981 +0000 UTC m=+2584.074755303" Oct 30 18:44:45.099434 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:45.099398 5232 patch_prober.go:29] interesting pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Oct 30 18:44:45.099632 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:45.099451 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" podUID="8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Oct 30 18:44:45.150552 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:45.150494 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:46.099448 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:46.099409 5232 patch_prober.go:29] interesting pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" start-of-body= Oct 30 18:44:46.099660 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:46.099466 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" podUID="8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.16:15021/healthz/ready\": dial tcp 10.133.1.16:15021: connect: connection refused" Oct 30 18:44:47.222593 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:47.222557 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:47.223321 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:47.222891 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:47.223998 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:47.223974 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:44:54.025335 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:54.025301 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp"] Oct 30 18:44:54.037166 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:54.037141 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-6frhp"] Oct 30 18:44:54.150934 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:54.150904 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:54.163732 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:54.163703 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:54.793970 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:54.793934 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:44:54.800489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:54.800464 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="992364e7-f08d-4aac-a893-8c91f303948f" path="/var/lib/kubelet/pods/992364e7-f08d-4aac-a893-8c91f303948f/volumes" Oct 30 18:44:55.778342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:55.778309 5232 scope.go:122] "RemoveContainer" containerID="c4409d9e541bc3de5b8d197e1f0e0d968174d1bcb4408601f3649dcddebea997" Oct 30 18:44:55.787717 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:55.787691 5232 scope.go:122] "RemoveContainer" containerID="9a093530a235dbb4c3a548ce644c90c1534787e6e54dd2b5b882d495d1c910c9" Oct 30 18:44:55.795772 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:44:55.795747 5232 scope.go:122] "RemoveContainer" containerID="d2a05dfc26e6de78228c34007426a77c6ab2ecc4a6346099d9ffe27efd7b09ac" Oct 30 18:44:57.033211 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:44:57.033170 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:45:00.136056 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:00.136011 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc"] Oct 30 18:45:00.141566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:00.141535 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc" Oct 30 18:45:00.161572 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:00.161538 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc"] Oct 30 18:45:00.171432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:00.171403 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:45:00.209869 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:00.209830 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kzmcx\" (UniqueName: \"kubernetes.io/projected/7b4bfa81-6cce-4187-8a3f-98508b727f55-kube-api-access-kzmcx\") pod \"maas-api-key-cleanup-29889765-2xwjc\" (UID: \"7b4bfa81-6cce-4187-8a3f-98508b727f55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc" Oct 30 18:45:00.210074 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:00.209896 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4bfa81-6cce-4187-8a3f-98508b727f55-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889765-2xwjc\" (UID: \"7b4bfa81-6cce-4187-8a3f-98508b727f55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc" Oct 30 18:45:00.311150 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:00.311111 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kzmcx\" (UniqueName: \"kubernetes.io/projected/7b4bfa81-6cce-4187-8a3f-98508b727f55-kube-api-access-kzmcx\") pod \"maas-api-key-cleanup-29889765-2xwjc\" (UID: \"7b4bfa81-6cce-4187-8a3f-98508b727f55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc" Oct 30 18:45:00.311345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:00.311192 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4bfa81-6cce-4187-8a3f-98508b727f55-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889765-2xwjc\" (UID: \"7b4bfa81-6cce-4187-8a3f-98508b727f55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc" Oct 30 18:45:00.311930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:00.311908 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4bfa81-6cce-4187-8a3f-98508b727f55-maas-api-service-ca\") pod \"maas-api-key-cleanup-29889765-2xwjc\" (UID: \"7b4bfa81-6cce-4187-8a3f-98508b727f55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc" Oct 30 18:45:00.330727 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:00.330690 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kzmcx\" (UniqueName: \"kubernetes.io/projected/7b4bfa81-6cce-4187-8a3f-98508b727f55-kube-api-access-kzmcx\") pod \"maas-api-key-cleanup-29889765-2xwjc\" (UID: \"7b4bfa81-6cce-4187-8a3f-98508b727f55\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc" Oct 30 18:45:00.455303 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:00.455222 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc" Oct 30 18:45:00.742993 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:00.742964 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc"] Oct 30 18:45:00.745131 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:45:00.745104 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b4bfa81_6cce_4187_8a3f_98508b727f55.slice/crio-817ab1f9ccbc7854d66124bed3bb18089688a5a269d5e37347185fbb2ad456b5 WatchSource:0}: Error finding container 817ab1f9ccbc7854d66124bed3bb18089688a5a269d5e37347185fbb2ad456b5: Status 404 returned error can't find the container with id 817ab1f9ccbc7854d66124bed3bb18089688a5a269d5e37347185fbb2ad456b5 Oct 30 18:45:00.814003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:00.813970 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc" event={"ID":"7b4bfa81-6cce-4187-8a3f-98508b727f55","Type":"ContainerStarted","Data":"817ab1f9ccbc7854d66124bed3bb18089688a5a269d5e37347185fbb2ad456b5"} Oct 30 18:45:01.819699 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:01.819600 5232 generic.go:356] "Generic (PLEG): container finished" podID="7b4bfa81-6cce-4187-8a3f-98508b727f55" containerID="4e98e54927c188476be92d82d805b2b70584ed8fe4480bd3af38bf9c5b5865e6" exitCode=60 Oct 30 18:45:01.820090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:01.819697 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc" event={"ID":"7b4bfa81-6cce-4187-8a3f-98508b727f55","Type":"ContainerDied","Data":"4e98e54927c188476be92d82d805b2b70584ed8fe4480bd3af38bf9c5b5865e6"} Oct 30 18:45:01.820090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:01.819882 5232 scope.go:122] "RemoveContainer" containerID="4e98e54927c188476be92d82d805b2b70584ed8fe4480bd3af38bf9c5b5865e6" Oct 30 18:45:02.826092 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:02.826058 5232 generic.go:356] "Generic (PLEG): container finished" podID="7b4bfa81-6cce-4187-8a3f-98508b727f55" containerID="946e341e0874a108a616d44b3ecdb47084eef4e48ab3ec8be71f553dee33ea27" exitCode=60 Oct 30 18:45:02.826579 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:02.826145 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc" event={"ID":"7b4bfa81-6cce-4187-8a3f-98508b727f55","Type":"ContainerDied","Data":"946e341e0874a108a616d44b3ecdb47084eef4e48ab3ec8be71f553dee33ea27"} Oct 30 18:45:02.826579 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:02.826206 5232 scope.go:122] "RemoveContainer" containerID="4e98e54927c188476be92d82d805b2b70584ed8fe4480bd3af38bf9c5b5865e6" Oct 30 18:45:02.826579 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:02.826392 5232 scope.go:122] "RemoveContainer" containerID="946e341e0874a108a616d44b3ecdb47084eef4e48ab3ec8be71f553dee33ea27" Oct 30 18:45:02.826767 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:45:02.826696 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889765-2xwjc_odh-ai-gateway-infra(7b4bfa81-6cce-4187-8a3f-98508b727f55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc" podUID="7b4bfa81-6cce-4187-8a3f-98508b727f55" Oct 30 18:45:03.833081 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:03.833053 5232 scope.go:122] "RemoveContainer" containerID="946e341e0874a108a616d44b3ecdb47084eef4e48ab3ec8be71f553dee33ea27" Oct 30 18:45:03.833477 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:45:03.833311 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29889765-2xwjc_odh-ai-gateway-infra(7b4bfa81-6cce-4187-8a3f-98508b727f55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc" podUID="7b4bfa81-6cce-4187-8a3f-98508b727f55" Oct 30 18:45:04.023131 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:04.023096 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct"] Oct 30 18:45:04.027757 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:04.027732 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-9cqct"] Oct 30 18:45:04.801614 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:04.801584 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d55fc27b-929d-4a6f-8930-ca3008ae0cd0" path="/var/lib/kubelet/pods/d55fc27b-929d-4a6f-8930-ca3008ae0cd0/volumes" Oct 30 18:45:17.795087 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:17.795049 5232 scope.go:122] "RemoveContainer" containerID="946e341e0874a108a616d44b3ecdb47084eef4e48ab3ec8be71f553dee33ea27" Oct 30 18:45:18.903462 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:18.903378 5232 generic.go:356] "Generic (PLEG): container finished" podID="7b4bfa81-6cce-4187-8a3f-98508b727f55" containerID="28e40b4bf28425895fdc1c423089e58a8f1570d9af6ec72c5ff89a0812f08271" exitCode=60 Oct 30 18:45:18.903462 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:18.903445 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc" event={"ID":"7b4bfa81-6cce-4187-8a3f-98508b727f55","Type":"ContainerDied","Data":"28e40b4bf28425895fdc1c423089e58a8f1570d9af6ec72c5ff89a0812f08271"} Oct 30 18:45:18.904003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:18.903501 5232 scope.go:122] "RemoveContainer" containerID="946e341e0874a108a616d44b3ecdb47084eef4e48ab3ec8be71f553dee33ea27" Oct 30 18:45:18.904003 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:18.903829 5232 scope.go:122] "RemoveContainer" containerID="28e40b4bf28425895fdc1c423089e58a8f1570d9af6ec72c5ff89a0812f08271" Oct 30 18:45:18.904092 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:45:18.904045 5232 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29889765-2xwjc_odh-ai-gateway-infra(7b4bfa81-6cce-4187-8a3f-98508b727f55)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc" podUID="7b4bfa81-6cce-4187-8a3f-98508b727f55" Oct 30 18:45:19.949787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:19.949757 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc"] Oct 30 18:45:20.085042 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:20.085019 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc" Oct 30 18:45:20.104498 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:20.104460 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b4bfa81-6cce-4187-8a3f-98508b727f55-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4bfa81-6cce-4187-8a3f-98508b727f55-maas-api-service-ca\") pod \"7b4bfa81-6cce-4187-8a3f-98508b727f55\" (UID: \"7b4bfa81-6cce-4187-8a3f-98508b727f55\") " Oct 30 18:45:20.104640 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:20.104584 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b4bfa81-6cce-4187-8a3f-98508b727f55-kube-api-access-kzmcx\" (UniqueName: \"kubernetes.io/projected/7b4bfa81-6cce-4187-8a3f-98508b727f55-kube-api-access-kzmcx\") pod \"7b4bfa81-6cce-4187-8a3f-98508b727f55\" (UID: \"7b4bfa81-6cce-4187-8a3f-98508b727f55\") " Oct 30 18:45:20.104789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:20.104770 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b4bfa81-6cce-4187-8a3f-98508b727f55-maas-api-service-ca" pod "7b4bfa81-6cce-4187-8a3f-98508b727f55" (UID: "7b4bfa81-6cce-4187-8a3f-98508b727f55"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:20.106745 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:20.106721 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b4bfa81-6cce-4187-8a3f-98508b727f55-kube-api-access-kzmcx" pod "7b4bfa81-6cce-4187-8a3f-98508b727f55" (UID: "7b4bfa81-6cce-4187-8a3f-98508b727f55"). InnerVolumeSpecName "kube-api-access-kzmcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:20.205871 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:20.205794 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b4bfa81-6cce-4187-8a3f-98508b727f55-maas-api-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:20.205871 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:20.205823 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kzmcx\" (UniqueName: \"kubernetes.io/projected/7b4bfa81-6cce-4187-8a3f-98508b727f55-kube-api-access-kzmcx\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:20.916484 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:20.916449 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc" Oct 30 18:45:20.916484 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:20.916468 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc" event={"ID":"7b4bfa81-6cce-4187-8a3f-98508b727f55","Type":"ContainerDied","Data":"817ab1f9ccbc7854d66124bed3bb18089688a5a269d5e37347185fbb2ad456b5"} Oct 30 18:45:20.916771 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:20.916551 5232 scope.go:122] "RemoveContainer" containerID="28e40b4bf28425895fdc1c423089e58a8f1570d9af6ec72c5ff89a0812f08271" Oct 30 18:45:20.968472 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:20.968446 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc"] Oct 30 18:45:20.980571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:20.980541 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29889765-2xwjc"] Oct 30 18:45:22.800587 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:22.800553 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b4bfa81-6cce-4187-8a3f-98508b727f55" path="/var/lib/kubelet/pods/7b4bfa81-6cce-4187-8a3f-98508b727f55/volumes" Oct 30 18:45:27.033452 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:45:27.033413 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:45:30.479278 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:30.479244 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr"] Oct 30 18:45:36.215265 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:36.215224 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph"] Oct 30 18:45:36.248152 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:36.248122 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2"] Oct 30 18:45:36.905962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:36.905917 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln"] Oct 30 18:45:36.906184 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:36.906142 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" podUID="d5ef8aad-dce5-41ca-826d-7fedab6368b4" containerName="maas-api" containerID="cri-o://7bd4c3af7262e49fa914e9e1cee11eaf6bff2802a7438d41560dcd3427899999" gracePeriod=30 Oct 30 18:45:38.156756 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:38.156728 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" Oct 30 18:45:38.286546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:38.286448 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5ef8aad-dce5-41ca-826d-7fedab6368b4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ef8aad-dce5-41ca-826d-7fedab6368b4-maas-api-tls\") pod \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\" (UID: \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\") " Oct 30 18:45:38.286718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:38.286600 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d5ef8aad-dce5-41ca-826d-7fedab6368b4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ef8aad-dce5-41ca-826d-7fedab6368b4-openshift-service-ca\") pod \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\" (UID: \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\") " Oct 30 18:45:38.286718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:38.286682 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d5ef8aad-dce5-41ca-826d-7fedab6368b4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ef8aad-dce5-41ca-826d-7fedab6368b4-metrics-certs\") pod \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\" (UID: \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\") " Oct 30 18:45:38.286839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:38.286719 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d5ef8aad-dce5-41ca-826d-7fedab6368b4-kube-api-access-ksj8l\" (UniqueName: \"kubernetes.io/projected/d5ef8aad-dce5-41ca-826d-7fedab6368b4-kube-api-access-ksj8l\") pod \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\" (UID: \"d5ef8aad-dce5-41ca-826d-7fedab6368b4\") " Oct 30 18:45:38.287004 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:38.286975 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ef8aad-dce5-41ca-826d-7fedab6368b4-openshift-service-ca" pod "d5ef8aad-dce5-41ca-826d-7fedab6368b4" (UID: "d5ef8aad-dce5-41ca-826d-7fedab6368b4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:38.288789 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:38.288758 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5ef8aad-dce5-41ca-826d-7fedab6368b4-maas-api-tls" pod "d5ef8aad-dce5-41ca-826d-7fedab6368b4" (UID: "d5ef8aad-dce5-41ca-826d-7fedab6368b4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:38.288927 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:38.288902 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d5ef8aad-dce5-41ca-826d-7fedab6368b4-metrics-certs" pod "d5ef8aad-dce5-41ca-826d-7fedab6368b4" (UID: "d5ef8aad-dce5-41ca-826d-7fedab6368b4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:38.288986 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:38.288938 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ef8aad-dce5-41ca-826d-7fedab6368b4-kube-api-access-ksj8l" pod "d5ef8aad-dce5-41ca-826d-7fedab6368b4" (UID: "d5ef8aad-dce5-41ca-826d-7fedab6368b4"). InnerVolumeSpecName "kube-api-access-ksj8l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:38.388047 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:38.388009 5232 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d5ef8aad-dce5-41ca-826d-7fedab6368b4-openshift-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:38.388047 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:38.388042 5232 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d5ef8aad-dce5-41ca-826d-7fedab6368b4-metrics-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:38.388047 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:38.388052 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksj8l\" (UniqueName: \"kubernetes.io/projected/d5ef8aad-dce5-41ca-826d-7fedab6368b4-kube-api-access-ksj8l\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:38.388295 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:38.388062 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d5ef8aad-dce5-41ca-826d-7fedab6368b4-maas-api-tls\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:39.013455 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:39.013413 5232 generic.go:356] "Generic (PLEG): container finished" podID="d5ef8aad-dce5-41ca-826d-7fedab6368b4" containerID="7bd4c3af7262e49fa914e9e1cee11eaf6bff2802a7438d41560dcd3427899999" exitCode=0 Oct 30 18:45:39.013684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:39.013465 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" Oct 30 18:45:39.013684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:39.013497 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" event={"ID":"d5ef8aad-dce5-41ca-826d-7fedab6368b4","Type":"ContainerDied","Data":"7bd4c3af7262e49fa914e9e1cee11eaf6bff2802a7438d41560dcd3427899999"} Oct 30 18:45:39.013684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:39.013575 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln" event={"ID":"d5ef8aad-dce5-41ca-826d-7fedab6368b4","Type":"ContainerDied","Data":"c486001ac2945cbae81c891023a1fe853617eb2a6d7e77ac4a7c8127fe749945"} Oct 30 18:45:39.013684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:39.013606 5232 scope.go:122] "RemoveContainer" containerID="7bd4c3af7262e49fa914e9e1cee11eaf6bff2802a7438d41560dcd3427899999" Oct 30 18:45:39.022609 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:39.022590 5232 scope.go:122] "RemoveContainer" containerID="7bd4c3af7262e49fa914e9e1cee11eaf6bff2802a7438d41560dcd3427899999" Oct 30 18:45:39.022885 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:45:39.022865 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd4c3af7262e49fa914e9e1cee11eaf6bff2802a7438d41560dcd3427899999\": container with ID starting with 7bd4c3af7262e49fa914e9e1cee11eaf6bff2802a7438d41560dcd3427899999 not found: ID does not exist" containerID="7bd4c3af7262e49fa914e9e1cee11eaf6bff2802a7438d41560dcd3427899999" Oct 30 18:45:39.022963 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:39.022898 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd4c3af7262e49fa914e9e1cee11eaf6bff2802a7438d41560dcd3427899999"} err="failed to get container status \"7bd4c3af7262e49fa914e9e1cee11eaf6bff2802a7438d41560dcd3427899999\": rpc error: code = NotFound desc = could not find container \"7bd4c3af7262e49fa914e9e1cee11eaf6bff2802a7438d41560dcd3427899999\": container with ID starting with 7bd4c3af7262e49fa914e9e1cee11eaf6bff2802a7438d41560dcd3427899999 not found: ID does not exist" Oct 30 18:45:39.066266 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:39.066188 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln"] Oct 30 18:45:39.078567 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:39.078534 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ddcab1-6b74474cf9-67sln"] Oct 30 18:45:40.801657 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:40.801620 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d5ef8aad-dce5-41ca-826d-7fedab6368b4" path="/var/lib/kubelet/pods/d5ef8aad-dce5-41ca-826d-7fedab6368b4/volumes" Oct 30 18:45:41.216676 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:41.216548 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" podUID="71533836-7a75-4c67-a826-4106480146e5" containerName="payload-processing" containerID="cri-o://a0f91980d800797315281eb4bc363cce5f6fbaa60ed05028b09ed6aa2686f28a" gracePeriod=25 Oct 30 18:45:41.248605 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:41.248557 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" podUID="61d1f62d-61a9-45b0-84f9-cb182481db06" containerName="payload-pre-processing" containerID="cri-o://7bd46686e4321e333df4728e9884fa036a62bcb9cbd511632c07a2e6cc2c1ee3" gracePeriod=25 Oct 30 18:45:41.638562 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:41.638497 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" podUID="71533836-7a75-4c67-a826-4106480146e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:45:41.638773 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:41.638743 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" podUID="61d1f62d-61a9-45b0-84f9-cb182481db06" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:45:45.499594 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.499551 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" podUID="261c1e8a-47e6-4ef3-8c59-fe509cc5d315" containerName="main" containerID="cri-o://96528e8913832022e8bbc626af7afd8d41fa0d28f0bde233db3150bbe8671420" gracePeriod=45 Oct 30 18:45:45.601072 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.601033 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs"] Oct 30 18:45:45.601315 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.601289 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" podUID="8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" containerName="istio-proxy" containerID="cri-o://340795e28946235c90b66e155f2a4f9e6197738699428ca00c9d85ce3023204c" gracePeriod=30 Oct 30 18:45:45.745498 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.745472 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:45:45.857695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.857604 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-tmp-dir\") pod \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " Oct 30 18:45:45.857695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.857673 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-model-cache\") pod \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " Oct 30 18:45:45.857695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.857693 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-home\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-home\") pod \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " Oct 30 18:45:45.858020 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.857729 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-kube-api-access-lmf5m\" (UniqueName: \"kubernetes.io/projected/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-kube-api-access-lmf5m\") pod \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " Oct 30 18:45:45.858020 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.857789 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-dshm\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-dshm\") pod \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " Oct 30 18:45:45.858020 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.857832 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-tls-certs\" (UniqueName: \"kubernetes.io/secret/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-tls-certs\") pod \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\" (UID: \"261c1e8a-47e6-4ef3-8c59-fe509cc5d315\") " Oct 30 18:45:45.858020 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.857911 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-tmp-dir" pod "261c1e8a-47e6-4ef3-8c59-fe509cc5d315" (UID: "261c1e8a-47e6-4ef3-8c59-fe509cc5d315"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:45:45.858020 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.857971 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-model-cache" pod "261c1e8a-47e6-4ef3-8c59-fe509cc5d315" (UID: "261c1e8a-47e6-4ef3-8c59-fe509cc5d315"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:45:45.858020 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.857985 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-home" pod "261c1e8a-47e6-4ef3-8c59-fe509cc5d315" (UID: "261c1e8a-47e6-4ef3-8c59-fe509cc5d315"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:45:45.858299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.858164 5232 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-tmp-dir\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:45.858299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.858185 5232 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-model-cache\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:45.858299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.858199 5232 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-home\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:45.860211 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.860181 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-tls-certs" pod "261c1e8a-47e6-4ef3-8c59-fe509cc5d315" (UID: "261c1e8a-47e6-4ef3-8c59-fe509cc5d315"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:45:45.860677 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.860652 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-dshm" pod "261c1e8a-47e6-4ef3-8c59-fe509cc5d315" (UID: "261c1e8a-47e6-4ef3-8c59-fe509cc5d315"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:45:45.860786 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.860676 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-kube-api-access-lmf5m" pod "261c1e8a-47e6-4ef3-8c59-fe509cc5d315" (UID: "261c1e8a-47e6-4ef3-8c59-fe509cc5d315"). InnerVolumeSpecName "kube-api-access-lmf5m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:45.959226 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.959185 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmf5m\" (UniqueName: \"kubernetes.io/projected/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-kube-api-access-lmf5m\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:45.959226 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.959216 5232 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-dshm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:45.959226 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:45.959229 5232 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/261c1e8a-47e6-4ef3-8c59-fe509cc5d315-tls-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:46.049290 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:46.049259 5232 generic.go:356] "Generic (PLEG): container finished" podID="261c1e8a-47e6-4ef3-8c59-fe509cc5d315" containerID="96528e8913832022e8bbc626af7afd8d41fa0d28f0bde233db3150bbe8671420" exitCode=0 Oct 30 18:45:46.049469 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:46.049308 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" Oct 30 18:45:46.049469 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:46.049313 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" event={"ID":"261c1e8a-47e6-4ef3-8c59-fe509cc5d315","Type":"ContainerDied","Data":"96528e8913832022e8bbc626af7afd8d41fa0d28f0bde233db3150bbe8671420"} Oct 30 18:45:46.049469 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:46.049418 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr" event={"ID":"261c1e8a-47e6-4ef3-8c59-fe509cc5d315","Type":"ContainerDied","Data":"9f002980aa477dd6d2eb91ae85550de466718346c57703c065813846d34c6895"} Oct 30 18:45:46.049469 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:46.049436 5232 scope.go:122] "RemoveContainer" containerID="96528e8913832022e8bbc626af7afd8d41fa0d28f0bde233db3150bbe8671420" Oct 30 18:45:46.058684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:46.058663 5232 scope.go:122] "RemoveContainer" containerID="96528e8913832022e8bbc626af7afd8d41fa0d28f0bde233db3150bbe8671420" Oct 30 18:45:46.058948 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:45:46.058928 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96528e8913832022e8bbc626af7afd8d41fa0d28f0bde233db3150bbe8671420\": container with ID starting with 96528e8913832022e8bbc626af7afd8d41fa0d28f0bde233db3150bbe8671420 not found: ID does not exist" containerID="96528e8913832022e8bbc626af7afd8d41fa0d28f0bde233db3150bbe8671420" Oct 30 18:45:46.058997 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:46.058957 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96528e8913832022e8bbc626af7afd8d41fa0d28f0bde233db3150bbe8671420"} err="failed to get container status \"96528e8913832022e8bbc626af7afd8d41fa0d28f0bde233db3150bbe8671420\": rpc error: code = NotFound desc = could not find container \"96528e8913832022e8bbc626af7afd8d41fa0d28f0bde233db3150bbe8671420\": container with ID starting with 96528e8913832022e8bbc626af7afd8d41fa0d28f0bde233db3150bbe8671420 not found: ID does not exist" Oct 30 18:45:46.101121 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:46.101092 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr"] Oct 30 18:45:46.106546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:46.106160 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-single-main-ddcab1/simulator-single-main-ddcab1-kserve-668fc5466c-695xr"] Oct 30 18:45:46.801704 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:46.801671 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="261c1e8a-47e6-4ef3-8c59-fe509cc5d315" path="/var/lib/kubelet/pods/261c1e8a-47e6-4ef3-8c59-fe509cc5d315/volumes" Oct 30 18:45:47.225555 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:47.225457 5232 patch_prober.go:29] interesting pod/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:45:47.225716 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:47.225576 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" podUID="8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:45:49.739353 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.739316 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8"] Oct 30 18:45:49.739787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.739760 5232 cpu_manager.go:414] "containerMap: removing container" podUID="261c1e8a-47e6-4ef3-8c59-fe509cc5d315" containerName="main" Oct 30 18:45:49.739787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.739772 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="261c1e8a-47e6-4ef3-8c59-fe509cc5d315" containerName="main" Oct 30 18:45:49.739787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.739782 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7b4bfa81-6cce-4187-8a3f-98508b727f55" containerName="cleanup" Oct 30 18:45:49.739787 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.739787 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4bfa81-6cce-4187-8a3f-98508b727f55" containerName="cleanup" Oct 30 18:45:49.739932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.739798 5232 cpu_manager.go:414] "containerMap: removing container" podUID="d5ef8aad-dce5-41ca-826d-7fedab6368b4" containerName="maas-api" Oct 30 18:45:49.739932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.739803 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d5ef8aad-dce5-41ca-826d-7fedab6368b4" containerName="maas-api" Oct 30 18:45:49.739932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.739809 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7b4bfa81-6cce-4187-8a3f-98508b727f55" containerName="cleanup" Oct 30 18:45:49.739932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.739814 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4bfa81-6cce-4187-8a3f-98508b727f55" containerName="cleanup" Oct 30 18:45:49.739932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.739888 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="d5ef8aad-dce5-41ca-826d-7fedab6368b4" containerName="maas-api" Oct 30 18:45:49.739932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.739895 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4bfa81-6cce-4187-8a3f-98508b727f55" containerName="cleanup" Oct 30 18:45:49.739932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.739903 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4bfa81-6cce-4187-8a3f-98508b727f55" containerName="cleanup" Oct 30 18:45:49.739932 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.739913 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="261c1e8a-47e6-4ef3-8c59-fe509cc5d315" containerName="main" Oct 30 18:45:49.744212 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.744192 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:49.764330 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.764302 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8"] Oct 30 18:45:49.817320 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.817285 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-c4c39655-kserve-self-signed-certs\"" Oct 30 18:45:49.899195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.899159 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-model-cache\") pod \"e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:49.899195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.899196 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5058d180-d75f-43a7-ba97-dc2421b9c59c-tls-certs\") pod \"e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:49.899430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.899226 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6wvt\" (UniqueName: \"kubernetes.io/projected/5058d180-d75f-43a7-ba97-dc2421b9c59c-kube-api-access-x6wvt\") pod \"e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:49.899430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.899343 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-dshm\") pod \"e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:49.899430 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.899371 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-tmp-dir\") pod \"e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:49.899597 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:49.899521 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-home\") pod \"e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:50.000811 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:50.000728 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-home\") pod \"e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:50.000811 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:50.000774 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-model-cache\") pod \"e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:50.000811 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:50.000799 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5058d180-d75f-43a7-ba97-dc2421b9c59c-tls-certs\") pod \"e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:50.001121 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:50.000824 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6wvt\" (UniqueName: \"kubernetes.io/projected/5058d180-d75f-43a7-ba97-dc2421b9c59c-kube-api-access-x6wvt\") pod \"e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:50.001121 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:50.000859 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-dshm\") pod \"e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:50.001121 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:50.000888 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-tmp-dir\") pod \"e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:50.001297 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:50.001205 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-home\") pod \"e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:50.001297 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:50.001245 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-model-cache\") pod \"e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:50.001297 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:50.001257 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-tmp-dir\") pod \"e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:50.003268 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:50.003244 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-dshm\") pod \"e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:50.003496 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:50.003478 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5058d180-d75f-43a7-ba97-dc2421b9c59c-tls-certs\") pod \"e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:50.019465 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:50.019442 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6wvt\" (UniqueName: \"kubernetes.io/projected/5058d180-d75f-43a7-ba97-dc2421b9c59c-kube-api-access-x6wvt\") pod \"e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:50.055494 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:50.055468 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:50.345019 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:50.344994 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8"] Oct 30 18:45:50.346993 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:45:50.346965 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5058d180_d75f_43a7_ba97_dc2421b9c59c.slice/crio-235d5b236d47a5354fefacdf279e912d7e55017cbb406dbcca88417827e65cae WatchSource:0}: Error finding container 235d5b236d47a5354fefacdf279e912d7e55017cbb406dbcca88417827e65cae: Status 404 returned error can't find the container with id 235d5b236d47a5354fefacdf279e912d7e55017cbb406dbcca88417827e65cae Oct 30 18:45:50.841774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:50.841748 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:45:51.010541 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.010482 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-podinfo\") pod \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " Oct 30 18:45:51.010663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.010598 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-data\") pod \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " Oct 30 18:45:51.010663 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.010635 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-credential-socket\") pod \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " Oct 30 18:45:51.010788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.010672 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-workload-certs\") pod \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " Oct 30 18:45:51.010788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.010720 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-workload-socket\") pod \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " Oct 30 18:45:51.010788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.010766 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-envoy\") pod \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " Oct 30 18:45:51.010951 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.010865 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istiod-ca-cert\") pod \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " Oct 30 18:45:51.010951 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.010915 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-kube-api-access-tm2ln\" (UniqueName: \"kubernetes.io/projected/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-kube-api-access-tm2ln\") pod \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " Oct 30 18:45:51.011063 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.010963 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-token\" (UniqueName: \"kubernetes.io/projected/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-token\") pod \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\" (UID: \"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25\") " Oct 30 18:45:51.011063 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.010956 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-data" pod "8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" (UID: "8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:45:51.011063 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.010973 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-credential-socket" pod "8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" (UID: "8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:45:51.011375 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.011295 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:51.011375 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.011319 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:51.011693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.011643 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-workload-certs" pod "8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" (UID: "8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:45:51.011822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.011785 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-workload-socket" pod "8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" (UID: "8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:45:51.012526 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.012478 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istiod-ca-cert" pod "8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" (UID: "8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:45:51.013450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.013414 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-podinfo" pod "8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" (UID: "8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:45:51.013571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.013534 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-envoy" pod "8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" (UID: "8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:45:51.014090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.014065 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-token" pod "8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" (UID: "8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:51.014090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.014079 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-kube-api-access-tm2ln" pod "8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" (UID: "8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25"). InnerVolumeSpecName "kube-api-access-tm2ln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:45:51.076865 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.076824 5232 generic.go:356] "Generic (PLEG): container finished" podID="8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" containerID="340795e28946235c90b66e155f2a4f9e6197738699428ca00c9d85ce3023204c" exitCode=0 Oct 30 18:45:51.077023 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.076862 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" event={"ID":"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25","Type":"ContainerDied","Data":"340795e28946235c90b66e155f2a4f9e6197738699428ca00c9d85ce3023204c"} Oct 30 18:45:51.077023 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.076876 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" Oct 30 18:45:51.077023 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.076923 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs" event={"ID":"8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25","Type":"ContainerDied","Data":"3cb58c0857e15299d3246736d15490384c4d21072564c04e100aa3f1af77b2a1"} Oct 30 18:45:51.077023 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.076948 5232 scope.go:122] "RemoveContainer" containerID="340795e28946235c90b66e155f2a4f9e6197738699428ca00c9d85ce3023204c" Oct 30 18:45:51.078645 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.078617 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" event={"ID":"5058d180-d75f-43a7-ba97-dc2421b9c59c","Type":"ContainerStarted","Data":"5b61b2b902cc041827d2d3d839a8a1d6ef7441a9262982aa3350633a01a90355"} Oct 30 18:45:51.078776 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.078657 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" event={"ID":"5058d180-d75f-43a7-ba97-dc2421b9c59c","Type":"ContainerStarted","Data":"235d5b236d47a5354fefacdf279e912d7e55017cbb406dbcca88417827e65cae"} Oct 30 18:45:51.091032 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.088369 5232 scope.go:122] "RemoveContainer" containerID="0fbaa748638e0361cff8ded0802098f8221691f98d304a33a288c267d4c1e321" Oct 30 18:45:51.097578 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.097558 5232 scope.go:122] "RemoveContainer" containerID="340795e28946235c90b66e155f2a4f9e6197738699428ca00c9d85ce3023204c" Oct 30 18:45:51.097826 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:45:51.097806 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"340795e28946235c90b66e155f2a4f9e6197738699428ca00c9d85ce3023204c\": container with ID starting with 340795e28946235c90b66e155f2a4f9e6197738699428ca00c9d85ce3023204c not found: ID does not exist" containerID="340795e28946235c90b66e155f2a4f9e6197738699428ca00c9d85ce3023204c" Oct 30 18:45:51.097891 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.097834 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"340795e28946235c90b66e155f2a4f9e6197738699428ca00c9d85ce3023204c"} err="failed to get container status \"340795e28946235c90b66e155f2a4f9e6197738699428ca00c9d85ce3023204c\": rpc error: code = NotFound desc = could not find container \"340795e28946235c90b66e155f2a4f9e6197738699428ca00c9d85ce3023204c\": container with ID starting with 340795e28946235c90b66e155f2a4f9e6197738699428ca00c9d85ce3023204c not found: ID does not exist" Oct 30 18:45:51.097891 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.097850 5232 scope.go:122] "RemoveContainer" containerID="0fbaa748638e0361cff8ded0802098f8221691f98d304a33a288c267d4c1e321" Oct 30 18:45:51.098130 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:45:51.098109 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0fbaa748638e0361cff8ded0802098f8221691f98d304a33a288c267d4c1e321\": container with ID starting with 0fbaa748638e0361cff8ded0802098f8221691f98d304a33a288c267d4c1e321 not found: ID does not exist" containerID="0fbaa748638e0361cff8ded0802098f8221691f98d304a33a288c267d4c1e321" Oct 30 18:45:51.098190 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.098139 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0fbaa748638e0361cff8ded0802098f8221691f98d304a33a288c267d4c1e321"} err="failed to get container status \"0fbaa748638e0361cff8ded0802098f8221691f98d304a33a288c267d4c1e321\": rpc error: code = NotFound desc = could not find container \"0fbaa748638e0361cff8ded0802098f8221691f98d304a33a288c267d4c1e321\": container with ID starting with 0fbaa748638e0361cff8ded0802098f8221691f98d304a33a288c267d4c1e321 not found: ID does not exist" Oct 30 18:45:51.112722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.112549 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:51.112722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.112576 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tm2ln\" (UniqueName: \"kubernetes.io/projected/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-kube-api-access-tm2ln\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:51.112722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.112592 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:51.112722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.112606 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:51.112722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.112617 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:51.112722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.112630 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:51.112722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.112642 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:45:51.132461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.132412 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" podStartSLOduration=2.132400751 podStartE2EDuration="2.132400751s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:45:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:45:51.130958292 +0000 UTC m=+2650.447378626" watchObservedRunningTime="2026-10-30 18:45:51.132400751 +0000 UTC m=+2650.448821429" Oct 30 18:45:51.168629 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.168604 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs"] Oct 30 18:45:51.173554 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.173498 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ddcab1-openshift-default-8fbbb88c7-p2svs"] Oct 30 18:45:51.638672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.638627 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" podUID="71533836-7a75-4c67-a826-4106480146e5" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:45:51.638867 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:51.638780 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" podUID="61d1f62d-61a9-45b0-84f9-cb182481db06" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 30 18:45:52.056464 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:52.056436 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:45:52.800822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:52.800793 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" path="/var/lib/kubelet/pods/8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25/volumes" Oct 30 18:45:55.824933 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:55.824896 5232 scope.go:122] "RemoveContainer" containerID="d6c753ae7c93cec5309f8ef36d442b519f219808091ef815bd3f27b23d7b044a" Oct 30 18:45:55.863064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:55.862642 5232 scope.go:122] "RemoveContainer" containerID="be1ec851f185699f10cb580af0180e9c0db238c7b187bcabfe417d00ad9a0867" Oct 30 18:45:55.901723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:45:55.901600 5232 scope.go:122] "RemoveContainer" containerID="397ec5bbcb82f4f0ad5a46ece52120f6cb21ef1898ccc4ed3990809c4c5d0f88" Oct 30 18:45:57.032989 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:45:57.032943 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:46:00.056127 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:00.056088 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:46:00.069217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:00.069188 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:46:00.153422 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:00.153387 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:46:01.494729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:01.494704 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" Oct 30 18:46:01.520028 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:01.520004 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" Oct 30 18:46:01.555921 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:01.555887 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/61d1f62d-61a9-45b0-84f9-cb182481db06-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d1f62d-61a9-45b0-84f9-cb182481db06-plugins-config-volume\") pod \"61d1f62d-61a9-45b0-84f9-cb182481db06\" (UID: \"61d1f62d-61a9-45b0-84f9-cb182481db06\") " Oct 30 18:46:01.556118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:01.555945 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71533836-7a75-4c67-a826-4106480146e5-kube-api-access-4dgbl\" (UniqueName: \"kubernetes.io/projected/71533836-7a75-4c67-a826-4106480146e5-kube-api-access-4dgbl\") pod \"71533836-7a75-4c67-a826-4106480146e5\" (UID: \"71533836-7a75-4c67-a826-4106480146e5\") " Oct 30 18:46:01.556118 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:01.555976 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/61d1f62d-61a9-45b0-84f9-cb182481db06-kube-api-access-g9vls\" (UniqueName: \"kubernetes.io/projected/61d1f62d-61a9-45b0-84f9-cb182481db06-kube-api-access-g9vls\") pod \"61d1f62d-61a9-45b0-84f9-cb182481db06\" (UID: \"61d1f62d-61a9-45b0-84f9-cb182481db06\") " Oct 30 18:46:01.556259 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:01.556153 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71533836-7a75-4c67-a826-4106480146e5-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71533836-7a75-4c67-a826-4106480146e5-plugins-config-volume\") pod \"71533836-7a75-4c67-a826-4106480146e5\" (UID: \"71533836-7a75-4c67-a826-4106480146e5\") " Oct 30 18:46:01.556367 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:01.556334 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/61d1f62d-61a9-45b0-84f9-cb182481db06-plugins-config-volume" pod "61d1f62d-61a9-45b0-84f9-cb182481db06" (UID: "61d1f62d-61a9-45b0-84f9-cb182481db06"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:01.556535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:01.556445 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71533836-7a75-4c67-a826-4106480146e5-plugins-config-volume" pod "71533836-7a75-4c67-a826-4106480146e5" (UID: "71533836-7a75-4c67-a826-4106480146e5"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:01.556615 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:01.556571 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/61d1f62d-61a9-45b0-84f9-cb182481db06-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:01.556615 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:01.556585 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/71533836-7a75-4c67-a826-4106480146e5-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:01.558433 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:01.558413 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71533836-7a75-4c67-a826-4106480146e5-kube-api-access-4dgbl" pod "71533836-7a75-4c67-a826-4106480146e5" (UID: "71533836-7a75-4c67-a826-4106480146e5"). InnerVolumeSpecName "kube-api-access-4dgbl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:01.558713 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:01.558696 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/61d1f62d-61a9-45b0-84f9-cb182481db06-kube-api-access-g9vls" pod "61d1f62d-61a9-45b0-84f9-cb182481db06" (UID: "61d1f62d-61a9-45b0-84f9-cb182481db06"). InnerVolumeSpecName "kube-api-access-g9vls". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:01.657013 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:01.656973 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dgbl\" (UniqueName: \"kubernetes.io/projected/71533836-7a75-4c67-a826-4106480146e5-kube-api-access-4dgbl\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:01.657013 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:01.657006 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9vls\" (UniqueName: \"kubernetes.io/projected/61d1f62d-61a9-45b0-84f9-cb182481db06-kube-api-access-g9vls\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:02.151835 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.151800 5232 generic.go:356] "Generic (PLEG): container finished" podID="61d1f62d-61a9-45b0-84f9-cb182481db06" containerID="7bd46686e4321e333df4728e9884fa036a62bcb9cbd511632c07a2e6cc2c1ee3" exitCode=0 Oct 30 18:46:02.152046 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.151848 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" Oct 30 18:46:02.152046 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.151890 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" event={"ID":"61d1f62d-61a9-45b0-84f9-cb182481db06","Type":"ContainerDied","Data":"7bd46686e4321e333df4728e9884fa036a62bcb9cbd511632c07a2e6cc2c1ee3"} Oct 30 18:46:02.152046 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.151952 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2" event={"ID":"61d1f62d-61a9-45b0-84f9-cb182481db06","Type":"ContainerDied","Data":"f39059c296c8dbe103c376af34f6708b25882784a2dc2e9e67d7f9f3f10f653b"} Oct 30 18:46:02.152046 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.151981 5232 scope.go:122] "RemoveContainer" containerID="7bd46686e4321e333df4728e9884fa036a62bcb9cbd511632c07a2e6cc2c1ee3" Oct 30 18:46:02.153300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.153276 5232 generic.go:356] "Generic (PLEG): container finished" podID="71533836-7a75-4c67-a826-4106480146e5" containerID="a0f91980d800797315281eb4bc363cce5f6fbaa60ed05028b09ed6aa2686f28a" exitCode=0 Oct 30 18:46:02.153436 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.153337 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" Oct 30 18:46:02.153436 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.153350 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" event={"ID":"71533836-7a75-4c67-a826-4106480146e5","Type":"ContainerDied","Data":"a0f91980d800797315281eb4bc363cce5f6fbaa60ed05028b09ed6aa2686f28a"} Oct 30 18:46:02.153436 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.153388 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph" event={"ID":"71533836-7a75-4c67-a826-4106480146e5","Type":"ContainerDied","Data":"54a6186da6ab7ede1099613de207b067f2a87d3e30f0aab6b5714821bf899669"} Oct 30 18:46:02.161207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.161184 5232 scope.go:122] "RemoveContainer" containerID="7bd46686e4321e333df4728e9884fa036a62bcb9cbd511632c07a2e6cc2c1ee3" Oct 30 18:46:02.161579 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:46:02.161558 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7bd46686e4321e333df4728e9884fa036a62bcb9cbd511632c07a2e6cc2c1ee3\": container with ID starting with 7bd46686e4321e333df4728e9884fa036a62bcb9cbd511632c07a2e6cc2c1ee3 not found: ID does not exist" containerID="7bd46686e4321e333df4728e9884fa036a62bcb9cbd511632c07a2e6cc2c1ee3" Oct 30 18:46:02.161671 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.161587 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7bd46686e4321e333df4728e9884fa036a62bcb9cbd511632c07a2e6cc2c1ee3"} err="failed to get container status \"7bd46686e4321e333df4728e9884fa036a62bcb9cbd511632c07a2e6cc2c1ee3\": rpc error: code = NotFound desc = could not find container \"7bd46686e4321e333df4728e9884fa036a62bcb9cbd511632c07a2e6cc2c1ee3\": container with ID starting with 7bd46686e4321e333df4728e9884fa036a62bcb9cbd511632c07a2e6cc2c1ee3 not found: ID does not exist" Oct 30 18:46:02.161671 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.161605 5232 scope.go:122] "RemoveContainer" containerID="a0f91980d800797315281eb4bc363cce5f6fbaa60ed05028b09ed6aa2686f28a" Oct 30 18:46:02.170371 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.170354 5232 scope.go:122] "RemoveContainer" containerID="a0f91980d800797315281eb4bc363cce5f6fbaa60ed05028b09ed6aa2686f28a" Oct 30 18:46:02.170644 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:46:02.170628 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a0f91980d800797315281eb4bc363cce5f6fbaa60ed05028b09ed6aa2686f28a\": container with ID starting with a0f91980d800797315281eb4bc363cce5f6fbaa60ed05028b09ed6aa2686f28a not found: ID does not exist" containerID="a0f91980d800797315281eb4bc363cce5f6fbaa60ed05028b09ed6aa2686f28a" Oct 30 18:46:02.170700 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.170652 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a0f91980d800797315281eb4bc363cce5f6fbaa60ed05028b09ed6aa2686f28a"} err="failed to get container status \"a0f91980d800797315281eb4bc363cce5f6fbaa60ed05028b09ed6aa2686f28a\": rpc error: code = NotFound desc = could not find container \"a0f91980d800797315281eb4bc363cce5f6fbaa60ed05028b09ed6aa2686f28a\": container with ID starting with a0f91980d800797315281eb4bc363cce5f6fbaa60ed05028b09ed6aa2686f28a not found: ID does not exist" Oct 30 18:46:02.201920 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.201892 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph"] Oct 30 18:46:02.203619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.203597 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-ddcab1-d4bcc5758rlxph"] Oct 30 18:46:02.238576 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.238551 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2"] Oct 30 18:46:02.242992 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.242969 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-ddcab1-68cbcxvfg2"] Oct 30 18:46:02.801529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.801484 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="61d1f62d-61a9-45b0-84f9-cb182481db06" path="/var/lib/kubelet/pods/61d1f62d-61a9-45b0-84f9-cb182481db06/volumes" Oct 30 18:46:02.801924 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:02.801870 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71533836-7a75-4c67-a826-4106480146e5" path="/var/lib/kubelet/pods/71533836-7a75-4c67-a826-4106480146e5/volumes" Oct 30 18:46:06.766998 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.766962 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m"] Oct 30 18:46:06.767393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.767337 5232 cpu_manager.go:414] "containerMap: removing container" podUID="71533836-7a75-4c67-a826-4106480146e5" containerName="payload-processing" Oct 30 18:46:06.767393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.767350 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71533836-7a75-4c67-a826-4106480146e5" containerName="payload-processing" Oct 30 18:46:06.767393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.767361 5232 cpu_manager.go:414] "containerMap: removing container" podUID="61d1f62d-61a9-45b0-84f9-cb182481db06" containerName="payload-pre-processing" Oct 30 18:46:06.767393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.767366 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="61d1f62d-61a9-45b0-84f9-cb182481db06" containerName="payload-pre-processing" Oct 30 18:46:06.767393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.767391 5232 cpu_manager.go:414] "containerMap: removing container" podUID="8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" containerName="istio-proxy" Oct 30 18:46:06.767393 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.767396 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" containerName="istio-proxy" Oct 30 18:46:06.767698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.767407 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7b4bfa81-6cce-4187-8a3f-98508b727f55" containerName="cleanup" Oct 30 18:46:06.767698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.767412 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b4bfa81-6cce-4187-8a3f-98508b727f55" containerName="cleanup" Oct 30 18:46:06.767698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.767423 5232 cpu_manager.go:414] "containerMap: removing container" podUID="8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" containerName="istio-proxy" Oct 30 18:46:06.767698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.767427 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" containerName="istio-proxy" Oct 30 18:46:06.767698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.767479 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" containerName="istio-proxy" Oct 30 18:46:06.767698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.767486 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b4bfa81-6cce-4187-8a3f-98508b727f55" containerName="cleanup" Oct 30 18:46:06.767698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.767493 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="71533836-7a75-4c67-a826-4106480146e5" containerName="payload-processing" Oct 30 18:46:06.767698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.767500 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="61d1f62d-61a9-45b0-84f9-cb182481db06" containerName="payload-pre-processing" Oct 30 18:46:06.767698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.767525 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="8ec1dda5-2fcf-444a-b6c2-40ac0e64ba25" containerName="istio-proxy" Oct 30 18:46:06.772203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.772179 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:06.791244 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.791193 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-c4c39655-kserve-self-signed-certs\"" Oct 30 18:46:06.793074 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.793046 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m"] Oct 30 18:46:06.799664 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.799638 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-model-cache\") pod \"e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:06.799801 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.799675 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62464e8e-7740-4d53-be21-5523bfab496b-tls-certs\") pod \"e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:06.799841 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.799790 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzw8k\" (UniqueName: \"kubernetes.io/projected/62464e8e-7740-4d53-be21-5523bfab496b-kube-api-access-wzw8k\") pod \"e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:06.799884 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.799842 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-dshm\") pod \"e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:06.799930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.799916 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-tmp-dir\") pod \"e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:06.799982 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.799971 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-home\") pod \"e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:06.900417 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.900384 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-tmp-dir\") pod \"e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:06.900645 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.900441 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-home\") pod \"e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:06.900645 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.900594 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-model-cache\") pod \"e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:06.900791 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.900661 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62464e8e-7740-4d53-be21-5523bfab496b-tls-certs\") pod \"e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:06.900791 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.900725 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzw8k\" (UniqueName: \"kubernetes.io/projected/62464e8e-7740-4d53-be21-5523bfab496b-kube-api-access-wzw8k\") pod \"e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:06.900791 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.900771 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-dshm\") pod \"e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:06.900960 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.900849 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-tmp-dir\") pod \"e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:06.901024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.900997 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-model-cache\") pod \"e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:06.901082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.901046 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-home\") pod \"e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:06.903092 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.903069 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-dshm\") pod \"e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:06.903445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.903423 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62464e8e-7740-4d53-be21-5523bfab496b-tls-certs\") pod \"e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:06.923416 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:06.923389 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzw8k\" (UniqueName: \"kubernetes.io/projected/62464e8e-7740-4d53-be21-5523bfab496b-kube-api-access-wzw8k\") pod \"e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:07.087414 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:07.087326 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:07.377858 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:07.377833 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m"] Oct 30 18:46:07.382388 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:46:07.382353 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod62464e8e_7740_4d53_be21_5523bfab496b.slice/crio-262b3c160fa0357cffbc652c7f69101b6d51324437fde42cd8b01326039cfe25 WatchSource:0}: Error finding container 262b3c160fa0357cffbc652c7f69101b6d51324437fde42cd8b01326039cfe25: Status 404 returned error can't find the container with id 262b3c160fa0357cffbc652c7f69101b6d51324437fde42cd8b01326039cfe25 Oct 30 18:46:08.184805 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:08.184768 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" event={"ID":"62464e8e-7740-4d53-be21-5523bfab496b","Type":"ContainerStarted","Data":"e3f137e83b8ec092cd168f7a47de742c0fd0060394077ea05776c0a979c163fe"} Oct 30 18:46:08.185233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:08.184811 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" event={"ID":"62464e8e-7740-4d53-be21-5523bfab496b","Type":"ContainerStarted","Data":"262b3c160fa0357cffbc652c7f69101b6d51324437fde42cd8b01326039cfe25"} Oct 30 18:46:08.225427 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:08.225360 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" podStartSLOduration=2.225343739 podStartE2EDuration="2.225343739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:46:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:46:08.223293241 +0000 UTC m=+2667.539713563" watchObservedRunningTime="2026-10-30 18:46:08.225343739 +0000 UTC m=+2667.541764063" Oct 30 18:46:09.087695 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:09.087661 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:17.087545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:17.087490 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:17.099820 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:17.099795 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:17.246346 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:17.246315 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:23.480468 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:23.480429 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj"] Oct 30 18:46:23.484247 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:23.484226 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj" Oct 30 18:46:23.502297 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:23.502068 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:46:23.511924 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:23.511899 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:46:23.513451 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:23.513431 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj"] Oct 30 18:46:23.590011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:23.589978 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v8r7k\" (UniqueName: \"kubernetes.io/projected/98e02b7e-baca-4304-a15e-9db6ae881dee-kube-api-access-v8r7k\") pod \"maas-api-revoke-sub-452b6d7b-ztvlj\" (UID: \"98e02b7e-baca-4304-a15e-9db6ae881dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj" Oct 30 18:46:23.590188 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:23.590016 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e02b7e-baca-4304-a15e-9db6ae881dee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-ztvlj\" (UID: \"98e02b7e-baca-4304-a15e-9db6ae881dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj" Oct 30 18:46:23.590188 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:23.590094 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e02b7e-baca-4304-a15e-9db6ae881dee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-ztvlj\" (UID: \"98e02b7e-baca-4304-a15e-9db6ae881dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj" Oct 30 18:46:23.690988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:23.690955 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v8r7k\" (UniqueName: \"kubernetes.io/projected/98e02b7e-baca-4304-a15e-9db6ae881dee-kube-api-access-v8r7k\") pod \"maas-api-revoke-sub-452b6d7b-ztvlj\" (UID: \"98e02b7e-baca-4304-a15e-9db6ae881dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj" Oct 30 18:46:23.690988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:23.690994 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e02b7e-baca-4304-a15e-9db6ae881dee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-ztvlj\" (UID: \"98e02b7e-baca-4304-a15e-9db6ae881dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj" Oct 30 18:46:23.691205 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:23.691019 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e02b7e-baca-4304-a15e-9db6ae881dee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-ztvlj\" (UID: \"98e02b7e-baca-4304-a15e-9db6ae881dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj" Oct 30 18:46:23.691655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:23.691625 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e02b7e-baca-4304-a15e-9db6ae881dee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-ztvlj\" (UID: \"98e02b7e-baca-4304-a15e-9db6ae881dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj" Oct 30 18:46:23.693435 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:23.693412 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e02b7e-baca-4304-a15e-9db6ae881dee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-ztvlj\" (UID: \"98e02b7e-baca-4304-a15e-9db6ae881dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj" Oct 30 18:46:23.711357 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:23.711327 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v8r7k\" (UniqueName: \"kubernetes.io/projected/98e02b7e-baca-4304-a15e-9db6ae881dee-kube-api-access-v8r7k\") pod \"maas-api-revoke-sub-452b6d7b-ztvlj\" (UID: \"98e02b7e-baca-4304-a15e-9db6ae881dee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj" Oct 30 18:46:23.797354 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:23.797322 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj" Oct 30 18:46:24.084964 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:24.084791 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj"] Oct 30 18:46:24.087180 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:46:24.087151 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98e02b7e_baca_4304_a15e_9db6ae881dee.slice/crio-5066c36a6e167353fb931013eece5b359325f96c2e73c63ff2b0e666ab496376 WatchSource:0}: Error finding container 5066c36a6e167353fb931013eece5b359325f96c2e73c63ff2b0e666ab496376: Status 404 returned error can't find the container with id 5066c36a6e167353fb931013eece5b359325f96c2e73c63ff2b0e666ab496376 Oct 30 18:46:24.267860 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:24.267824 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj" event={"ID":"98e02b7e-baca-4304-a15e-9db6ae881dee","Type":"ContainerStarted","Data":"5066c36a6e167353fb931013eece5b359325f96c2e73c63ff2b0e666ab496376"} Oct 30 18:46:25.273613 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:25.273580 5232 generic.go:356] "Generic (PLEG): container finished" podID="98e02b7e-baca-4304-a15e-9db6ae881dee" containerID="fa28903c1ca9d8662a8f23f1d5649a58860d6b8248677c35c9e41235f6e5afaa" exitCode=0 Oct 30 18:46:25.274024 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:25.273623 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj" event={"ID":"98e02b7e-baca-4304-a15e-9db6ae881dee","Type":"ContainerDied","Data":"fa28903c1ca9d8662a8f23f1d5649a58860d6b8248677c35c9e41235f6e5afaa"} Oct 30 18:46:26.474882 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:26.474847 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj" Oct 30 18:46:26.619231 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:26.619130 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/98e02b7e-baca-4304-a15e-9db6ae881dee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e02b7e-baca-4304-a15e-9db6ae881dee-maas-api-xks-ca\") pod \"98e02b7e-baca-4304-a15e-9db6ae881dee\" (UID: \"98e02b7e-baca-4304-a15e-9db6ae881dee\") " Oct 30 18:46:26.619231 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:26.619227 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98e02b7e-baca-4304-a15e-9db6ae881dee-kube-api-access-v8r7k\" (UniqueName: \"kubernetes.io/projected/98e02b7e-baca-4304-a15e-9db6ae881dee-kube-api-access-v8r7k\") pod \"98e02b7e-baca-4304-a15e-9db6ae881dee\" (UID: \"98e02b7e-baca-4304-a15e-9db6ae881dee\") " Oct 30 18:46:26.619495 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:26.619314 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98e02b7e-baca-4304-a15e-9db6ae881dee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e02b7e-baca-4304-a15e-9db6ae881dee-maas-api-openshift-ca\") pod \"98e02b7e-baca-4304-a15e-9db6ae881dee\" (UID: \"98e02b7e-baca-4304-a15e-9db6ae881dee\") " Oct 30 18:46:26.620001 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:26.619969 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98e02b7e-baca-4304-a15e-9db6ae881dee-maas-api-openshift-ca" pod "98e02b7e-baca-4304-a15e-9db6ae881dee" (UID: "98e02b7e-baca-4304-a15e-9db6ae881dee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:26.622142 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:26.622074 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/98e02b7e-baca-4304-a15e-9db6ae881dee-maas-api-xks-ca" pod "98e02b7e-baca-4304-a15e-9db6ae881dee" (UID: "98e02b7e-baca-4304-a15e-9db6ae881dee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:26.622279 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:26.622217 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98e02b7e-baca-4304-a15e-9db6ae881dee-kube-api-access-v8r7k" pod "98e02b7e-baca-4304-a15e-9db6ae881dee" (UID: "98e02b7e-baca-4304-a15e-9db6ae881dee"). InnerVolumeSpecName "kube-api-access-v8r7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:26.720807 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:26.720759 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/98e02b7e-baca-4304-a15e-9db6ae881dee-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:26.720807 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:26.720801 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v8r7k\" (UniqueName: \"kubernetes.io/projected/98e02b7e-baca-4304-a15e-9db6ae881dee-kube-api-access-v8r7k\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:26.720807 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:26.720818 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/98e02b7e-baca-4304-a15e-9db6ae881dee-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:27.159816 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:46:27.159732 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:46:27.304533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:27.304071 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj" event={"ID":"98e02b7e-baca-4304-a15e-9db6ae881dee","Type":"ContainerDied","Data":"5066c36a6e167353fb931013eece5b359325f96c2e73c63ff2b0e666ab496376"} Oct 30 18:46:27.304533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:27.304139 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5066c36a6e167353fb931013eece5b359325f96c2e73c63ff2b0e666ab496376" Oct 30 18:46:27.304533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:27.304207 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj" Oct 30 18:46:34.715238 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:34.715198 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8"] Oct 30 18:46:40.639761 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:40.639726 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m"] Oct 30 18:46:44.840000 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:44.839964 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn"] Oct 30 18:46:44.840390 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:44.840302 5232 cpu_manager.go:414] "containerMap: removing container" podUID="98e02b7e-baca-4304-a15e-9db6ae881dee" containerName="revoke-keys" Oct 30 18:46:44.840390 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:44.840312 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98e02b7e-baca-4304-a15e-9db6ae881dee" containerName="revoke-keys" Oct 30 18:46:44.840469 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:44.840393 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="98e02b7e-baca-4304-a15e-9db6ae881dee" containerName="revoke-keys" Oct 30 18:46:44.843544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:44.843529 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn" Oct 30 18:46:44.860163 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:44.860129 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn"] Oct 30 18:46:44.870763 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:44.870740 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:46:44.870763 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:44.870740 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:46:44.885216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:44.885188 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1b778a-57bc-42c7-8568-f94b7b2eb631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-h75jn\" (UID: \"5d1b778a-57bc-42c7-8568-f94b7b2eb631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn" Oct 30 18:46:44.885355 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:44.885261 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1b778a-57bc-42c7-8568-f94b7b2eb631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-h75jn\" (UID: \"5d1b778a-57bc-42c7-8568-f94b7b2eb631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn" Oct 30 18:46:44.885402 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:44.885356 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jpvvn\" (UniqueName: \"kubernetes.io/projected/5d1b778a-57bc-42c7-8568-f94b7b2eb631-kube-api-access-jpvvn\") pod \"maas-api-revoke-sub-3e31a506-h75jn\" (UID: \"5d1b778a-57bc-42c7-8568-f94b7b2eb631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn" Oct 30 18:46:44.986382 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:44.986344 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1b778a-57bc-42c7-8568-f94b7b2eb631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-h75jn\" (UID: \"5d1b778a-57bc-42c7-8568-f94b7b2eb631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn" Oct 30 18:46:44.986570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:44.986441 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1b778a-57bc-42c7-8568-f94b7b2eb631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-h75jn\" (UID: \"5d1b778a-57bc-42c7-8568-f94b7b2eb631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn" Oct 30 18:46:44.986570 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:44.986477 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jpvvn\" (UniqueName: \"kubernetes.io/projected/5d1b778a-57bc-42c7-8568-f94b7b2eb631-kube-api-access-jpvvn\") pod \"maas-api-revoke-sub-3e31a506-h75jn\" (UID: \"5d1b778a-57bc-42c7-8568-f94b7b2eb631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn" Oct 30 18:46:44.987220 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:44.987143 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1b778a-57bc-42c7-8568-f94b7b2eb631-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-h75jn\" (UID: \"5d1b778a-57bc-42c7-8568-f94b7b2eb631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn" Oct 30 18:46:44.988832 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:44.988810 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1b778a-57bc-42c7-8568-f94b7b2eb631-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-h75jn\" (UID: \"5d1b778a-57bc-42c7-8568-f94b7b2eb631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn" Oct 30 18:46:45.005639 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:45.005613 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jpvvn\" (UniqueName: \"kubernetes.io/projected/5d1b778a-57bc-42c7-8568-f94b7b2eb631-kube-api-access-jpvvn\") pod \"maas-api-revoke-sub-3e31a506-h75jn\" (UID: \"5d1b778a-57bc-42c7-8568-f94b7b2eb631\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn" Oct 30 18:46:45.156705 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:45.156625 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn" Oct 30 18:46:45.447855 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:45.447831 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn"] Oct 30 18:46:45.450275 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:46:45.450237 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5d1b778a_57bc_42c7_8568_f94b7b2eb631.slice/crio-cddf1088bcd17823b63cfc596f34c92ca97200500aa592d87255959714b544f5 WatchSource:0}: Error finding container cddf1088bcd17823b63cfc596f34c92ca97200500aa592d87255959714b544f5: Status 404 returned error can't find the container with id cddf1088bcd17823b63cfc596f34c92ca97200500aa592d87255959714b544f5 Oct 30 18:46:46.422961 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:46.422916 5232 generic.go:356] "Generic (PLEG): container finished" podID="5d1b778a-57bc-42c7-8568-f94b7b2eb631" containerID="d915320c905cedc0664885a4edf5b835d219848c796a4d8ecd7c8d162d333003" exitCode=0 Oct 30 18:46:46.423479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:46.423027 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn" event={"ID":"5d1b778a-57bc-42c7-8568-f94b7b2eb631","Type":"ContainerDied","Data":"d915320c905cedc0664885a4edf5b835d219848c796a4d8ecd7c8d162d333003"} Oct 30 18:46:46.423479 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:46.423147 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn" event={"ID":"5d1b778a-57bc-42c7-8568-f94b7b2eb631","Type":"ContainerStarted","Data":"cddf1088bcd17823b63cfc596f34c92ca97200500aa592d87255959714b544f5"} Oct 30 18:46:47.994812 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:47.994757 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn" Oct 30 18:46:48.106588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:48.106217 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5d1b778a-57bc-42c7-8568-f94b7b2eb631-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1b778a-57bc-42c7-8568-f94b7b2eb631-maas-api-xks-ca\") pod \"5d1b778a-57bc-42c7-8568-f94b7b2eb631\" (UID: \"5d1b778a-57bc-42c7-8568-f94b7b2eb631\") " Oct 30 18:46:48.106588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:48.106319 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5d1b778a-57bc-42c7-8568-f94b7b2eb631-kube-api-access-jpvvn\" (UniqueName: \"kubernetes.io/projected/5d1b778a-57bc-42c7-8568-f94b7b2eb631-kube-api-access-jpvvn\") pod \"5d1b778a-57bc-42c7-8568-f94b7b2eb631\" (UID: \"5d1b778a-57bc-42c7-8568-f94b7b2eb631\") " Oct 30 18:46:48.106588 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:48.106383 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5d1b778a-57bc-42c7-8568-f94b7b2eb631-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1b778a-57bc-42c7-8568-f94b7b2eb631-maas-api-openshift-ca\") pod \"5d1b778a-57bc-42c7-8568-f94b7b2eb631\" (UID: \"5d1b778a-57bc-42c7-8568-f94b7b2eb631\") " Oct 30 18:46:48.107178 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:48.107150 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5d1b778a-57bc-42c7-8568-f94b7b2eb631-maas-api-openshift-ca" pod "5d1b778a-57bc-42c7-8568-f94b7b2eb631" (UID: "5d1b778a-57bc-42c7-8568-f94b7b2eb631"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:46:48.185734 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:48.185670 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5d1b778a-57bc-42c7-8568-f94b7b2eb631-maas-api-xks-ca" pod "5d1b778a-57bc-42c7-8568-f94b7b2eb631" (UID: "5d1b778a-57bc-42c7-8568-f94b7b2eb631"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:48.186530 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:48.186465 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5d1b778a-57bc-42c7-8568-f94b7b2eb631-kube-api-access-jpvvn" pod "5d1b778a-57bc-42c7-8568-f94b7b2eb631" (UID: "5d1b778a-57bc-42c7-8568-f94b7b2eb631"). InnerVolumeSpecName "kube-api-access-jpvvn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:48.209242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:48.209156 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5d1b778a-57bc-42c7-8568-f94b7b2eb631-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:48.209242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:48.209194 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jpvvn\" (UniqueName: \"kubernetes.io/projected/5d1b778a-57bc-42c7-8568-f94b7b2eb631-kube-api-access-jpvvn\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:48.209242 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:48.209210 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5d1b778a-57bc-42c7-8568-f94b7b2eb631-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:48.454590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:48.451962 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn" event={"ID":"5d1b778a-57bc-42c7-8568-f94b7b2eb631","Type":"ContainerDied","Data":"cddf1088bcd17823b63cfc596f34c92ca97200500aa592d87255959714b544f5"} Oct 30 18:46:48.454590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:48.452029 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cddf1088bcd17823b63cfc596f34c92ca97200500aa592d87255959714b544f5" Oct 30 18:46:48.454590 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:48.452104 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn" Oct 30 18:46:49.735424 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:49.735386 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" podUID="5058d180-d75f-43a7-ba97-dc2421b9c59c" containerName="main" containerID="cri-o://5b61b2b902cc041827d2d3d839a8a1d6ef7441a9262982aa3350633a01a90355" gracePeriod=45 Oct 30 18:46:49.986743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:49.986717 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:46:50.131129 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.131094 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-tmp-dir\") pod \"5058d180-d75f-43a7-ba97-dc2421b9c59c\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " Oct 30 18:46:50.131317 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.131148 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-dshm\") pod \"5058d180-d75f-43a7-ba97-dc2421b9c59c\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " Oct 30 18:46:50.131317 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.131218 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-home\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-home\") pod \"5058d180-d75f-43a7-ba97-dc2421b9c59c\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " Oct 30 18:46:50.131317 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.131249 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5058d180-d75f-43a7-ba97-dc2421b9c59c-tls-certs\" (UniqueName: \"kubernetes.io/secret/5058d180-d75f-43a7-ba97-dc2421b9c59c-tls-certs\") pod \"5058d180-d75f-43a7-ba97-dc2421b9c59c\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " Oct 30 18:46:50.131317 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.131300 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5058d180-d75f-43a7-ba97-dc2421b9c59c-kube-api-access-x6wvt\" (UniqueName: \"kubernetes.io/projected/5058d180-d75f-43a7-ba97-dc2421b9c59c-kube-api-access-x6wvt\") pod \"5058d180-d75f-43a7-ba97-dc2421b9c59c\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " Oct 30 18:46:50.131580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.131329 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-model-cache\") pod \"5058d180-d75f-43a7-ba97-dc2421b9c59c\" (UID: \"5058d180-d75f-43a7-ba97-dc2421b9c59c\") " Oct 30 18:46:50.131580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.131363 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-tmp-dir" pod "5058d180-d75f-43a7-ba97-dc2421b9c59c" (UID: "5058d180-d75f-43a7-ba97-dc2421b9c59c"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:46:50.131580 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.131553 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-home" pod "5058d180-d75f-43a7-ba97-dc2421b9c59c" (UID: "5058d180-d75f-43a7-ba97-dc2421b9c59c"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:46:50.131753 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.131683 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-model-cache" pod "5058d180-d75f-43a7-ba97-dc2421b9c59c" (UID: "5058d180-d75f-43a7-ba97-dc2421b9c59c"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:46:50.131753 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.131713 5232 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-tmp-dir\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:50.131753 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.131727 5232 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-home\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:50.133761 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.133738 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5058d180-d75f-43a7-ba97-dc2421b9c59c-kube-api-access-x6wvt" pod "5058d180-d75f-43a7-ba97-dc2421b9c59c" (UID: "5058d180-d75f-43a7-ba97-dc2421b9c59c"). InnerVolumeSpecName "kube-api-access-x6wvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:50.133857 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.133778 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-dshm" pod "5058d180-d75f-43a7-ba97-dc2421b9c59c" (UID: "5058d180-d75f-43a7-ba97-dc2421b9c59c"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:46:50.133895 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.133878 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5058d180-d75f-43a7-ba97-dc2421b9c59c-tls-certs" pod "5058d180-d75f-43a7-ba97-dc2421b9c59c" (UID: "5058d180-d75f-43a7-ba97-dc2421b9c59c"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:50.233075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.233049 5232 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5058d180-d75f-43a7-ba97-dc2421b9c59c-tls-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:50.233075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.233074 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6wvt\" (UniqueName: \"kubernetes.io/projected/5058d180-d75f-43a7-ba97-dc2421b9c59c-kube-api-access-x6wvt\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:50.233269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.233084 5232 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-model-cache\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:50.233269 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.233094 5232 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5058d180-d75f-43a7-ba97-dc2421b9c59c-dshm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:50.461413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.461322 5232 generic.go:356] "Generic (PLEG): container finished" podID="5058d180-d75f-43a7-ba97-dc2421b9c59c" containerID="5b61b2b902cc041827d2d3d839a8a1d6ef7441a9262982aa3350633a01a90355" exitCode=0 Oct 30 18:46:50.461413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.461363 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" Oct 30 18:46:50.461413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.461398 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" event={"ID":"5058d180-d75f-43a7-ba97-dc2421b9c59c","Type":"ContainerDied","Data":"5b61b2b902cc041827d2d3d839a8a1d6ef7441a9262982aa3350633a01a90355"} Oct 30 18:46:50.461709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.461442 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8" event={"ID":"5058d180-d75f-43a7-ba97-dc2421b9c59c","Type":"ContainerDied","Data":"235d5b236d47a5354fefacdf279e912d7e55017cbb406dbcca88417827e65cae"} Oct 30 18:46:50.461709 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.461460 5232 scope.go:122] "RemoveContainer" containerID="5b61b2b902cc041827d2d3d839a8a1d6ef7441a9262982aa3350633a01a90355" Oct 30 18:46:50.470412 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.470397 5232 scope.go:122] "RemoveContainer" containerID="5b61b2b902cc041827d2d3d839a8a1d6ef7441a9262982aa3350633a01a90355" Oct 30 18:46:50.470680 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:46:50.470661 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5b61b2b902cc041827d2d3d839a8a1d6ef7441a9262982aa3350633a01a90355\": container with ID starting with 5b61b2b902cc041827d2d3d839a8a1d6ef7441a9262982aa3350633a01a90355 not found: ID does not exist" containerID="5b61b2b902cc041827d2d3d839a8a1d6ef7441a9262982aa3350633a01a90355" Oct 30 18:46:50.470749 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.470689 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5b61b2b902cc041827d2d3d839a8a1d6ef7441a9262982aa3350633a01a90355"} err="failed to get container status \"5b61b2b902cc041827d2d3d839a8a1d6ef7441a9262982aa3350633a01a90355\": rpc error: code = NotFound desc = could not find container \"5b61b2b902cc041827d2d3d839a8a1d6ef7441a9262982aa3350633a01a90355\": container with ID starting with 5b61b2b902cc041827d2d3d839a8a1d6ef7441a9262982aa3350633a01a90355 not found: ID does not exist" Oct 30 18:46:50.510810 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.510786 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8"] Oct 30 18:46:50.514320 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.514300 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-c4c39655-kserve-6d899465d5-mglk8"] Oct 30 18:46:50.800446 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:50.800417 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5058d180-d75f-43a7-ba97-dc2421b9c59c" path="/var/lib/kubelet/pods/5058d180-d75f-43a7-ba97-dc2421b9c59c/volumes" Oct 30 18:46:56.015560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.015528 5232 scope.go:122] "RemoveContainer" containerID="8975f10c895a5c2ed1aa80c0ae4a4faaedfe935fe80ec94cab0e51f72670f4e2" Oct 30 18:46:56.025203 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.025183 5232 scope.go:122] "RemoveContainer" containerID="7831fbafcdc71c8e928b4eeea027283ab14dad5b6995d5f0d4f9aa0f3ddd2f96" Oct 30 18:46:56.111889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.111823 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" podUID="62464e8e-7740-4d53-be21-5523bfab496b" containerName="main" containerID="cri-o://e3f137e83b8ec092cd168f7a47de742c0fd0060394077ea05776c0a979c163fe" gracePeriod=45 Oct 30 18:46:56.236362 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.236311 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" podUID="62464e8e-7740-4d53-be21-5523bfab496b" containerName="main" probeResult="failure" output="Get \"https://10.133.1.23:8000/ready\": dial tcp 10.133.1.23:8000: connect: connection refused" Oct 30 18:46:56.366368 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.366345 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:56.491299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.491267 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-dshm\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-dshm\") pod \"62464e8e-7740-4d53-be21-5523bfab496b\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " Oct 30 18:46:56.491490 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.491312 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-model-cache\") pod \"62464e8e-7740-4d53-be21-5523bfab496b\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " Oct 30 18:46:56.491490 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.491336 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-tmp-dir\") pod \"62464e8e-7740-4d53-be21-5523bfab496b\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " Oct 30 18:46:56.491490 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.491375 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62464e8e-7740-4d53-be21-5523bfab496b-kube-api-access-wzw8k\" (UniqueName: \"kubernetes.io/projected/62464e8e-7740-4d53-be21-5523bfab496b-kube-api-access-wzw8k\") pod \"62464e8e-7740-4d53-be21-5523bfab496b\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " Oct 30 18:46:56.491490 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.491398 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-home\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-home\") pod \"62464e8e-7740-4d53-be21-5523bfab496b\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " Oct 30 18:46:56.491490 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.491476 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/62464e8e-7740-4d53-be21-5523bfab496b-tls-certs\" (UniqueName: \"kubernetes.io/secret/62464e8e-7740-4d53-be21-5523bfab496b-tls-certs\") pod \"62464e8e-7740-4d53-be21-5523bfab496b\" (UID: \"62464e8e-7740-4d53-be21-5523bfab496b\") " Oct 30 18:46:56.491797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.491477 5232 generic.go:356] "Generic (PLEG): container finished" podID="62464e8e-7740-4d53-be21-5523bfab496b" containerID="e3f137e83b8ec092cd168f7a47de742c0fd0060394077ea05776c0a979c163fe" exitCode=0 Oct 30 18:46:56.491797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.491611 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" event={"ID":"62464e8e-7740-4d53-be21-5523bfab496b","Type":"ContainerDied","Data":"e3f137e83b8ec092cd168f7a47de742c0fd0060394077ea05776c0a979c163fe"} Oct 30 18:46:56.491797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.491653 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" event={"ID":"62464e8e-7740-4d53-be21-5523bfab496b","Type":"ContainerDied","Data":"262b3c160fa0357cffbc652c7f69101b6d51324437fde42cd8b01326039cfe25"} Oct 30 18:46:56.491797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.491660 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m" Oct 30 18:46:56.491797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.491679 5232 scope.go:122] "RemoveContainer" containerID="e3f137e83b8ec092cd168f7a47de742c0fd0060394077ea05776c0a979c163fe" Oct 30 18:46:56.491797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.491587 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-model-cache" pod "62464e8e-7740-4d53-be21-5523bfab496b" (UID: "62464e8e-7740-4d53-be21-5523bfab496b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:46:56.491797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.491667 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-tmp-dir" pod "62464e8e-7740-4d53-be21-5523bfab496b" (UID: "62464e8e-7740-4d53-be21-5523bfab496b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:46:56.491797 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.491747 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-home" pod "62464e8e-7740-4d53-be21-5523bfab496b" (UID: "62464e8e-7740-4d53-be21-5523bfab496b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:46:56.492186 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.491924 5232 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-model-cache\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:56.492186 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.491945 5232 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-tmp-dir\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:56.492186 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.491958 5232 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-home\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:56.494016 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.493982 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-dshm" pod "62464e8e-7740-4d53-be21-5523bfab496b" (UID: "62464e8e-7740-4d53-be21-5523bfab496b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:46:56.494016 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.494003 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62464e8e-7740-4d53-be21-5523bfab496b-kube-api-access-wzw8k" pod "62464e8e-7740-4d53-be21-5523bfab496b" (UID: "62464e8e-7740-4d53-be21-5523bfab496b"). InnerVolumeSpecName "kube-api-access-wzw8k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:46:56.494205 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.494130 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/62464e8e-7740-4d53-be21-5523bfab496b-tls-certs" pod "62464e8e-7740-4d53-be21-5523bfab496b" (UID: "62464e8e-7740-4d53-be21-5523bfab496b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:46:56.509956 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.509929 5232 scope.go:122] "RemoveContainer" containerID="e3f137e83b8ec092cd168f7a47de742c0fd0060394077ea05776c0a979c163fe" Oct 30 18:46:56.510216 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:46:56.510196 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e3f137e83b8ec092cd168f7a47de742c0fd0060394077ea05776c0a979c163fe\": container with ID starting with e3f137e83b8ec092cd168f7a47de742c0fd0060394077ea05776c0a979c163fe not found: ID does not exist" containerID="e3f137e83b8ec092cd168f7a47de742c0fd0060394077ea05776c0a979c163fe" Oct 30 18:46:56.510263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.510226 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e3f137e83b8ec092cd168f7a47de742c0fd0060394077ea05776c0a979c163fe"} err="failed to get container status \"e3f137e83b8ec092cd168f7a47de742c0fd0060394077ea05776c0a979c163fe\": rpc error: code = NotFound desc = could not find container \"e3f137e83b8ec092cd168f7a47de742c0fd0060394077ea05776c0a979c163fe\": container with ID starting with e3f137e83b8ec092cd168f7a47de742c0fd0060394077ea05776c0a979c163fe not found: ID does not exist" Oct 30 18:46:56.592917 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.592858 5232 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/62464e8e-7740-4d53-be21-5523bfab496b-dshm\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:56.592917 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.592883 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzw8k\" (UniqueName: \"kubernetes.io/projected/62464e8e-7740-4d53-be21-5523bfab496b-kube-api-access-wzw8k\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:56.592917 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.592892 5232 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/62464e8e-7740-4d53-be21-5523bfab496b-tls-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:46:56.835090 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.835061 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m"] Oct 30 18:46:56.842677 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:56.842638 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-c4c39655-kserve-5c4998bf45-kfr5m"] Oct 30 18:46:57.137605 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:46:57.136686 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:46:58.802625 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:46:58.802586 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62464e8e-7740-4d53-be21-5523bfab496b" path="/var/lib/kubelet/pods/62464e8e-7740-4d53-be21-5523bfab496b/volumes" Oct 30 18:47:04.025836 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:04.025804 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp"] Oct 30 18:47:04.037954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:04.037929 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-qsknp"] Oct 30 18:47:04.801283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:04.801246 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7044133a-08fc-4861-b938-941ea0a59b2d" path="/var/lib/kubelet/pods/7044133a-08fc-4861-b938-941ea0a59b2d/volumes" Oct 30 18:47:11.542300 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.542262 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22"] Oct 30 18:47:11.542822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.542666 5232 cpu_manager.go:414] "containerMap: removing container" podUID="62464e8e-7740-4d53-be21-5523bfab496b" containerName="main" Oct 30 18:47:11.542822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.542679 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62464e8e-7740-4d53-be21-5523bfab496b" containerName="main" Oct 30 18:47:11.542822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.542692 5232 cpu_manager.go:414] "containerMap: removing container" podUID="5d1b778a-57bc-42c7-8568-f94b7b2eb631" containerName="revoke-keys" Oct 30 18:47:11.542822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.542697 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5d1b778a-57bc-42c7-8568-f94b7b2eb631" containerName="revoke-keys" Oct 30 18:47:11.542822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.542711 5232 cpu_manager.go:414] "containerMap: removing container" podUID="5058d180-d75f-43a7-ba97-dc2421b9c59c" containerName="main" Oct 30 18:47:11.542822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.542720 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5058d180-d75f-43a7-ba97-dc2421b9c59c" containerName="main" Oct 30 18:47:11.542822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.542780 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="62464e8e-7740-4d53-be21-5523bfab496b" containerName="main" Oct 30 18:47:11.542822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.542792 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="5d1b778a-57bc-42c7-8568-f94b7b2eb631" containerName="revoke-keys" Oct 30 18:47:11.542822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.542802 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="5058d180-d75f-43a7-ba97-dc2421b9c59c" containerName="main" Oct 30 18:47:11.545904 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.545883 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22" Oct 30 18:47:11.572377 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.572354 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 30 18:47:11.572618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.572437 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-49gmg\"" Oct 30 18:47:11.574966 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.574941 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22"] Oct 30 18:47:11.695743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.695693 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0417a2b-5395-48a7-8a61-b7befdaea670-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24d8e614-9wq22\" (UID: \"e0417a2b-5395-48a7-8a61-b7befdaea670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22" Oct 30 18:47:11.695743 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.695749 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhmln\" (UniqueName: \"kubernetes.io/projected/e0417a2b-5395-48a7-8a61-b7befdaea670-kube-api-access-jhmln\") pod \"maas-api-revoke-sub-24d8e614-9wq22\" (UID: \"e0417a2b-5395-48a7-8a61-b7befdaea670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22" Oct 30 18:47:11.696002 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.695831 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0417a2b-5395-48a7-8a61-b7befdaea670-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24d8e614-9wq22\" (UID: \"e0417a2b-5395-48a7-8a61-b7befdaea670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22" Oct 30 18:47:11.796425 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.796324 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0417a2b-5395-48a7-8a61-b7befdaea670-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24d8e614-9wq22\" (UID: \"e0417a2b-5395-48a7-8a61-b7befdaea670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22" Oct 30 18:47:11.796425 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.796385 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhmln\" (UniqueName: \"kubernetes.io/projected/e0417a2b-5395-48a7-8a61-b7befdaea670-kube-api-access-jhmln\") pod \"maas-api-revoke-sub-24d8e614-9wq22\" (UID: \"e0417a2b-5395-48a7-8a61-b7befdaea670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22" Oct 30 18:47:11.796761 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.796467 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0417a2b-5395-48a7-8a61-b7befdaea670-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24d8e614-9wq22\" (UID: \"e0417a2b-5395-48a7-8a61-b7befdaea670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22" Oct 30 18:47:11.797014 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.796993 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0417a2b-5395-48a7-8a61-b7befdaea670-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-24d8e614-9wq22\" (UID: \"e0417a2b-5395-48a7-8a61-b7befdaea670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22" Oct 30 18:47:11.799229 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.799208 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0417a2b-5395-48a7-8a61-b7befdaea670-maas-api-xks-ca\") pod \"maas-api-revoke-sub-24d8e614-9wq22\" (UID: \"e0417a2b-5395-48a7-8a61-b7befdaea670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22" Oct 30 18:47:11.815135 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.815109 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhmln\" (UniqueName: \"kubernetes.io/projected/e0417a2b-5395-48a7-8a61-b7befdaea670-kube-api-access-jhmln\") pod \"maas-api-revoke-sub-24d8e614-9wq22\" (UID: \"e0417a2b-5395-48a7-8a61-b7befdaea670\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22" Oct 30 18:47:11.858272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:11.858248 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22" Oct 30 18:47:12.148545 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:12.148500 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22"] Oct 30 18:47:12.151220 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:47:12.151195 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode0417a2b_5395_48a7_8a61_b7befdaea670.slice/crio-346b5446414293fab9a23b5ef31ea940026dc3cf800484485503df65136079c5 WatchSource:0}: Error finding container 346b5446414293fab9a23b5ef31ea940026dc3cf800484485503df65136079c5: Status 404 returned error can't find the container with id 346b5446414293fab9a23b5ef31ea940026dc3cf800484485503df65136079c5 Oct 30 18:47:12.625718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:12.625681 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22" event={"ID":"e0417a2b-5395-48a7-8a61-b7befdaea670","Type":"ContainerStarted","Data":"620031e8c51b7eda0defecbdc9e981a655d275af1602228bd143c73d154b71f7"} Oct 30 18:47:12.626208 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:12.625739 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22" event={"ID":"e0417a2b-5395-48a7-8a61-b7befdaea670","Type":"ContainerStarted","Data":"346b5446414293fab9a23b5ef31ea940026dc3cf800484485503df65136079c5"} Oct 30 18:47:12.666341 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:12.666282 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22" podStartSLOduration=1.666269807 podStartE2EDuration="1.666269807s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:12.665057871 +0000 UTC m=+2731.981478192" watchObservedRunningTime="2026-10-30 18:47:12.666269807 +0000 UTC m=+2731.982690122" Oct 30 18:47:13.632703 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:13.632668 5232 generic.go:356] "Generic (PLEG): container finished" podID="e0417a2b-5395-48a7-8a61-b7befdaea670" containerID="620031e8c51b7eda0defecbdc9e981a655d275af1602228bd143c73d154b71f7" exitCode=0 Oct 30 18:47:13.633180 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:13.632746 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22" event={"ID":"e0417a2b-5395-48a7-8a61-b7befdaea670","Type":"ContainerDied","Data":"620031e8c51b7eda0defecbdc9e981a655d275af1602228bd143c73d154b71f7"} Oct 30 18:47:14.772000 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:14.771974 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22" Oct 30 18:47:14.924145 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:14.924072 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e0417a2b-5395-48a7-8a61-b7befdaea670-kube-api-access-jhmln\" (UniqueName: \"kubernetes.io/projected/e0417a2b-5395-48a7-8a61-b7befdaea670-kube-api-access-jhmln\") pod \"e0417a2b-5395-48a7-8a61-b7befdaea670\" (UID: \"e0417a2b-5395-48a7-8a61-b7befdaea670\") " Oct 30 18:47:14.924145 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:14.924105 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e0417a2b-5395-48a7-8a61-b7befdaea670-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0417a2b-5395-48a7-8a61-b7befdaea670-maas-api-openshift-ca\") pod \"e0417a2b-5395-48a7-8a61-b7befdaea670\" (UID: \"e0417a2b-5395-48a7-8a61-b7befdaea670\") " Oct 30 18:47:14.924342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:14.924149 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e0417a2b-5395-48a7-8a61-b7befdaea670-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0417a2b-5395-48a7-8a61-b7befdaea670-maas-api-xks-ca\") pod \"e0417a2b-5395-48a7-8a61-b7befdaea670\" (UID: \"e0417a2b-5395-48a7-8a61-b7befdaea670\") " Oct 30 18:47:14.924549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:14.924523 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e0417a2b-5395-48a7-8a61-b7befdaea670-maas-api-openshift-ca" pod "e0417a2b-5395-48a7-8a61-b7befdaea670" (UID: "e0417a2b-5395-48a7-8a61-b7befdaea670"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:14.926369 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:14.926341 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e0417a2b-5395-48a7-8a61-b7befdaea670-maas-api-xks-ca" pod "e0417a2b-5395-48a7-8a61-b7befdaea670" (UID: "e0417a2b-5395-48a7-8a61-b7befdaea670"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:14.926369 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:14.926347 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e0417a2b-5395-48a7-8a61-b7befdaea670-kube-api-access-jhmln" pod "e0417a2b-5395-48a7-8a61-b7befdaea670" (UID: "e0417a2b-5395-48a7-8a61-b7befdaea670"). InnerVolumeSpecName "kube-api-access-jhmln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:15.025280 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:15.025244 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhmln\" (UniqueName: \"kubernetes.io/projected/e0417a2b-5395-48a7-8a61-b7befdaea670-kube-api-access-jhmln\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:47:15.025280 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:15.025273 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e0417a2b-5395-48a7-8a61-b7befdaea670-maas-api-openshift-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:47:15.025471 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:15.025287 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e0417a2b-5395-48a7-8a61-b7befdaea670-maas-api-xks-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:47:15.643998 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:15.643968 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22" Oct 30 18:47:15.643998 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:15.643990 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22" event={"ID":"e0417a2b-5395-48a7-8a61-b7befdaea670","Type":"ContainerDied","Data":"346b5446414293fab9a23b5ef31ea940026dc3cf800484485503df65136079c5"} Oct 30 18:47:15.644235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:15.644041 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="346b5446414293fab9a23b5ef31ea940026dc3cf800484485503df65136079c5" Oct 30 18:47:23.966281 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:23.966245 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5"] Oct 30 18:47:23.966795 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:23.966608 5232 cpu_manager.go:414] "containerMap: removing container" podUID="e0417a2b-5395-48a7-8a61-b7befdaea670" containerName="revoke-keys" Oct 30 18:47:23.966795 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:23.966619 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e0417a2b-5395-48a7-8a61-b7befdaea670" containerName="revoke-keys" Oct 30 18:47:23.966795 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:23.966698 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="e0417a2b-5395-48a7-8a61-b7befdaea670" containerName="revoke-keys" Oct 30 18:47:23.969935 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:23.969912 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" Oct 30 18:47:23.983842 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:23.983815 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8"] Oct 30 18:47:23.987364 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:23.987344 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" Oct 30 18:47:23.996357 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:23.996332 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7zv2\" (UniqueName: \"kubernetes.io/projected/20bb7d71-4f14-41ba-b486-e3f7e44288d4-kube-api-access-g7zv2\") pod \"maas-api-78b5489d4d-rqcd5\" (UID: \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" Oct 30 18:47:23.996538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:23.996373 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20bb7d71-4f14-41ba-b486-e3f7e44288d4-openshift-service-ca\") pod \"maas-api-78b5489d4d-rqcd5\" (UID: \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" Oct 30 18:47:23.996538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:23.996413 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20bb7d71-4f14-41ba-b486-e3f7e44288d4-metrics-certs\") pod \"maas-api-78b5489d4d-rqcd5\" (UID: \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" Oct 30 18:47:23.996538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:23.996457 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20bb7d71-4f14-41ba-b486-e3f7e44288d4-maas-api-tls\") pod \"maas-api-78b5489d4d-rqcd5\" (UID: \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" Oct 30 18:47:23.997339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:23.997314 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5"] Oct 30 18:47:24.001810 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.001778 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-llpqx\"" Oct 30 18:47:24.001939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.001789 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 30 18:47:24.001939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.001872 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 30 18:47:24.010157 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.010127 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8"] Oct 30 18:47:24.097678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.097643 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v52cg\" (UniqueName: \"kubernetes.io/projected/8c447e9c-f711-4c11-9c6d-09a9c28f65a3-kube-api-access-v52cg\") pod \"maas-api-5fc75ddbcb-q62f8\" (UID: \"8c447e9c-f711-4c11-9c6d-09a9c28f65a3\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" Oct 30 18:47:24.097678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.097690 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c447e9c-f711-4c11-9c6d-09a9c28f65a3-maas-api-tls\") pod \"maas-api-5fc75ddbcb-q62f8\" (UID: \"8c447e9c-f711-4c11-9c6d-09a9c28f65a3\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" Oct 30 18:47:24.097946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.097723 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7zv2\" (UniqueName: \"kubernetes.io/projected/20bb7d71-4f14-41ba-b486-e3f7e44288d4-kube-api-access-g7zv2\") pod \"maas-api-78b5489d4d-rqcd5\" (UID: \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" Oct 30 18:47:24.097946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.097892 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20bb7d71-4f14-41ba-b486-e3f7e44288d4-openshift-service-ca\") pod \"maas-api-78b5489d4d-rqcd5\" (UID: \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" Oct 30 18:47:24.097946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.097913 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c447e9c-f711-4c11-9c6d-09a9c28f65a3-metrics-certs\") pod \"maas-api-5fc75ddbcb-q62f8\" (UID: \"8c447e9c-f711-4c11-9c6d-09a9c28f65a3\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" Oct 30 18:47:24.097946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.097937 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20bb7d71-4f14-41ba-b486-e3f7e44288d4-metrics-certs\") pod \"maas-api-78b5489d4d-rqcd5\" (UID: \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" Oct 30 18:47:24.098160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.097971 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20bb7d71-4f14-41ba-b486-e3f7e44288d4-maas-api-tls\") pod \"maas-api-78b5489d4d-rqcd5\" (UID: \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" Oct 30 18:47:24.098160 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.098052 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c447e9c-f711-4c11-9c6d-09a9c28f65a3-openshift-service-ca\") pod \"maas-api-5fc75ddbcb-q62f8\" (UID: \"8c447e9c-f711-4c11-9c6d-09a9c28f65a3\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" Oct 30 18:47:24.098557 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.098535 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20bb7d71-4f14-41ba-b486-e3f7e44288d4-openshift-service-ca\") pod \"maas-api-78b5489d4d-rqcd5\" (UID: \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" Oct 30 18:47:24.100538 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.100500 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20bb7d71-4f14-41ba-b486-e3f7e44288d4-metrics-certs\") pod \"maas-api-78b5489d4d-rqcd5\" (UID: \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" Oct 30 18:47:24.100790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.100771 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20bb7d71-4f14-41ba-b486-e3f7e44288d4-maas-api-tls\") pod \"maas-api-78b5489d4d-rqcd5\" (UID: \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" Oct 30 18:47:24.116272 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.116250 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7zv2\" (UniqueName: \"kubernetes.io/projected/20bb7d71-4f14-41ba-b486-e3f7e44288d4-kube-api-access-g7zv2\") pod \"maas-api-78b5489d4d-rqcd5\" (UID: \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\") " pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" Oct 30 18:47:24.199480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.199454 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c447e9c-f711-4c11-9c6d-09a9c28f65a3-openshift-service-ca\") pod \"maas-api-5fc75ddbcb-q62f8\" (UID: \"8c447e9c-f711-4c11-9c6d-09a9c28f65a3\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" Oct 30 18:47:24.199637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.199525 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v52cg\" (UniqueName: \"kubernetes.io/projected/8c447e9c-f711-4c11-9c6d-09a9c28f65a3-kube-api-access-v52cg\") pod \"maas-api-5fc75ddbcb-q62f8\" (UID: \"8c447e9c-f711-4c11-9c6d-09a9c28f65a3\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" Oct 30 18:47:24.199637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.199552 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c447e9c-f711-4c11-9c6d-09a9c28f65a3-maas-api-tls\") pod \"maas-api-5fc75ddbcb-q62f8\" (UID: \"8c447e9c-f711-4c11-9c6d-09a9c28f65a3\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" Oct 30 18:47:24.199637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.199582 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c447e9c-f711-4c11-9c6d-09a9c28f65a3-metrics-certs\") pod \"maas-api-5fc75ddbcb-q62f8\" (UID: \"8c447e9c-f711-4c11-9c6d-09a9c28f65a3\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" Oct 30 18:47:24.200105 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.200073 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8c447e9c-f711-4c11-9c6d-09a9c28f65a3-openshift-service-ca\") pod \"maas-api-5fc75ddbcb-q62f8\" (UID: \"8c447e9c-f711-4c11-9c6d-09a9c28f65a3\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" Oct 30 18:47:24.202126 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.202107 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8c447e9c-f711-4c11-9c6d-09a9c28f65a3-maas-api-tls\") pod \"maas-api-5fc75ddbcb-q62f8\" (UID: \"8c447e9c-f711-4c11-9c6d-09a9c28f65a3\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" Oct 30 18:47:24.202303 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.202279 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8c447e9c-f711-4c11-9c6d-09a9c28f65a3-metrics-certs\") pod \"maas-api-5fc75ddbcb-q62f8\" (UID: \"8c447e9c-f711-4c11-9c6d-09a9c28f65a3\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" Oct 30 18:47:24.218720 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.218665 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v52cg\" (UniqueName: \"kubernetes.io/projected/8c447e9c-f711-4c11-9c6d-09a9c28f65a3-kube-api-access-v52cg\") pod \"maas-api-5fc75ddbcb-q62f8\" (UID: \"8c447e9c-f711-4c11-9c6d-09a9c28f65a3\") " pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" Oct 30 18:47:24.285933 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.285908 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" Oct 30 18:47:24.303986 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.303965 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" Oct 30 18:47:24.585445 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.585418 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5"] Oct 30 18:47:24.587453 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:47:24.587426 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20bb7d71_4f14_41ba_b486_e3f7e44288d4.slice/crio-13c70fdbec07f28e61573384a1829379d03bad8672319089a861551318b08bc4 WatchSource:0}: Error finding container 13c70fdbec07f28e61573384a1829379d03bad8672319089a861551318b08bc4: Status 404 returned error can't find the container with id 13c70fdbec07f28e61573384a1829379d03bad8672319089a861551318b08bc4 Oct 30 18:47:24.588766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.588749 5232 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:47:24.616903 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.616883 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8"] Oct 30 18:47:24.618819 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:47:24.618793 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c447e9c_f711_4c11_9c6d_09a9c28f65a3.slice/crio-d3bf3ecf3b0a9d60ec4c2d436eed9901374f18baaacae2d80a04ea662afbda4d WatchSource:0}: Error finding container d3bf3ecf3b0a9d60ec4c2d436eed9901374f18baaacae2d80a04ea662afbda4d: Status 404 returned error can't find the container with id d3bf3ecf3b0a9d60ec4c2d436eed9901374f18baaacae2d80a04ea662afbda4d Oct 30 18:47:24.684546 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.684502 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" event={"ID":"8c447e9c-f711-4c11-9c6d-09a9c28f65a3","Type":"ContainerStarted","Data":"d3bf3ecf3b0a9d60ec4c2d436eed9901374f18baaacae2d80a04ea662afbda4d"} Oct 30 18:47:24.685766 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:24.685741 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" event={"ID":"20bb7d71-4f14-41ba-b486-e3f7e44288d4","Type":"ContainerStarted","Data":"13c70fdbec07f28e61573384a1829379d03bad8672319089a861551318b08bc4"} Oct 30 18:47:25.692298 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:25.692252 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" event={"ID":"8c447e9c-f711-4c11-9c6d-09a9c28f65a3","Type":"ContainerStarted","Data":"8061926b29eef42ee5220c52fa2e05ca08005bb5e86c9ff97d8a20cf887809d9"} Oct 30 18:47:25.692830 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:25.692338 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" Oct 30 18:47:25.694197 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:25.694175 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" event={"ID":"20bb7d71-4f14-41ba-b486-e3f7e44288d4","Type":"ContainerStarted","Data":"d80f1865979edfa596f07a94d96385edb66cffd70afa17a2d35a1f5543032ef0"} Oct 30 18:47:25.694342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:25.694295 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" Oct 30 18:47:25.733730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:25.733670 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" podStartSLOduration=2.4041198919999998 podStartE2EDuration="2.733658061s" totalImagesPullingTime="329.538169ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:25.731678259 +0000 UTC m=+2745.048098573" watchObservedRunningTime="2026-10-30 18:47:25.733658061 +0000 UTC m=+2745.050078431" Oct 30 18:47:25.770483 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:25.770427 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" podStartSLOduration=2.332634908 podStartE2EDuration="2.770413865s" totalImagesPullingTime="437.778957ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:25.770065252 +0000 UTC m=+2745.086485574" watchObservedRunningTime="2026-10-30 18:47:25.770413865 +0000 UTC m=+2745.086834180" Oct 30 18:47:27.032876 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:47:27.032841 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:47:31.214351 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.214308 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl"] Oct 30 18:47:31.218233 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.218207 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.235584 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.235555 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl"] Oct 30 18:47:31.245734 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.245708 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-069c47a6-openshift-default-dockercfg-6q76d\"" Oct 30 18:47:31.266566 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.266543 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-credential-socket\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.266742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.266574 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-workload-socket\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.266742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.266603 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istiod-ca-cert\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.266742 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.266719 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-data\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.266954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.266753 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-podinfo\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.266954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.266780 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k6624\" (UniqueName: \"kubernetes.io/projected/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-kube-api-access-k6624\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.266954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.266862 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-token\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.266954 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.266882 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-workload-certs\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.267196 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.266968 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-envoy\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.367808 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.367775 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-data\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.367808 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.367813 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-podinfo\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.368041 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.367845 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k6624\" (UniqueName: \"kubernetes.io/projected/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-kube-api-access-k6624\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.368041 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.367872 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-token\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.368041 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.368028 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-workload-certs\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.368197 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.368097 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-envoy\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.368197 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.368181 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-credential-socket\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.368316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.368207 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-workload-socket\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.368316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.368242 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-data\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.368316 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.368281 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istiod-ca-cert\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.368523 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.368396 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-workload-certs\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.368583 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.368531 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-workload-socket\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.368620 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.368601 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-credential-socket\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.369224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.369201 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istiod-ca-cert\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.370486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.370458 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-envoy\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.370684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.370661 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-podinfo\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.388774 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.388746 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-token\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.388879 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.388826 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k6624\" (UniqueName: \"kubernetes.io/projected/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-kube-api-access-k6624\") pod \"e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.531293 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.531253 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:31.704645 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.704617 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-5fc75ddbcb-q62f8" Oct 30 18:47:31.705051 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.705033 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" Oct 30 18:47:31.777475 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.777445 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5"] Oct 30 18:47:31.777691 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.777664 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" podUID="20bb7d71-4f14-41ba-b486-e3f7e44288d4" containerName="maas-api" containerID="cri-o://d80f1865979edfa596f07a94d96385edb66cffd70afa17a2d35a1f5543032ef0" gracePeriod=30 Oct 30 18:47:31.852651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.852624 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl"] Oct 30 18:47:31.855286 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:47:31.855253 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2dd47bb_3ad6_4e84_b4a9_a56c8655b98d.slice/crio-2b54b3ee07c873f7b57876c297b400e062e6bb924c266abe70b2edd6244634d0 WatchSource:0}: Error finding container 2b54b3ee07c873f7b57876c297b400e062e6bb924c266abe70b2edd6244634d0: Status 404 returned error can't find the container with id 2b54b3ee07c873f7b57876c297b400e062e6bb924c266abe70b2edd6244634d0 Oct 30 18:47:31.883587 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.883556 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:47:31.883682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.883631 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:47:31.883682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:31.883670 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:47:32.015005 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.014976 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" Oct 30 18:47:32.077026 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.076940 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20bb7d71-4f14-41ba-b486-e3f7e44288d4-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20bb7d71-4f14-41ba-b486-e3f7e44288d4-metrics-certs\") pod \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\" (UID: \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\") " Oct 30 18:47:32.077026 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.076975 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20bb7d71-4f14-41ba-b486-e3f7e44288d4-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20bb7d71-4f14-41ba-b486-e3f7e44288d4-openshift-service-ca\") pod \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\" (UID: \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\") " Oct 30 18:47:32.077026 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.077010 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20bb7d71-4f14-41ba-b486-e3f7e44288d4-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20bb7d71-4f14-41ba-b486-e3f7e44288d4-maas-api-tls\") pod \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\" (UID: \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\") " Oct 30 18:47:32.077317 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.077042 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20bb7d71-4f14-41ba-b486-e3f7e44288d4-kube-api-access-g7zv2\" (UniqueName: \"kubernetes.io/projected/20bb7d71-4f14-41ba-b486-e3f7e44288d4-kube-api-access-g7zv2\") pod \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\" (UID: \"20bb7d71-4f14-41ba-b486-e3f7e44288d4\") " Oct 30 18:47:32.077768 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.077725 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20bb7d71-4f14-41ba-b486-e3f7e44288d4-openshift-service-ca" pod "20bb7d71-4f14-41ba-b486-e3f7e44288d4" (UID: "20bb7d71-4f14-41ba-b486-e3f7e44288d4"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:47:32.080015 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.079975 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20bb7d71-4f14-41ba-b486-e3f7e44288d4-kube-api-access-g7zv2" pod "20bb7d71-4f14-41ba-b486-e3f7e44288d4" (UID: "20bb7d71-4f14-41ba-b486-e3f7e44288d4"). InnerVolumeSpecName "kube-api-access-g7zv2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:47:32.080155 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.080020 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20bb7d71-4f14-41ba-b486-e3f7e44288d4-metrics-certs" pod "20bb7d71-4f14-41ba-b486-e3f7e44288d4" (UID: "20bb7d71-4f14-41ba-b486-e3f7e44288d4"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:32.080155 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.080061 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20bb7d71-4f14-41ba-b486-e3f7e44288d4-maas-api-tls" pod "20bb7d71-4f14-41ba-b486-e3f7e44288d4" (UID: "20bb7d71-4f14-41ba-b486-e3f7e44288d4"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:47:32.177736 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.177702 5232 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20bb7d71-4f14-41ba-b486-e3f7e44288d4-metrics-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:47:32.177736 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.177733 5232 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20bb7d71-4f14-41ba-b486-e3f7e44288d4-openshift-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:47:32.177736 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.177742 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20bb7d71-4f14-41ba-b486-e3f7e44288d4-maas-api-tls\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:47:32.177979 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.177751 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7zv2\" (UniqueName: \"kubernetes.io/projected/20bb7d71-4f14-41ba-b486-e3f7e44288d4-kube-api-access-g7zv2\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:47:32.728791 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.728695 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" event={"ID":"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d","Type":"ContainerStarted","Data":"98bc08a7ab3ca8c3871fccdb5f691ccb5b71ca0377f5d0db8840f098e4989c3f"} Oct 30 18:47:32.728791 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.728757 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" event={"ID":"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d","Type":"ContainerStarted","Data":"2b54b3ee07c873f7b57876c297b400e062e6bb924c266abe70b2edd6244634d0"} Oct 30 18:47:32.729941 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.729917 5232 generic.go:356] "Generic (PLEG): container finished" podID="20bb7d71-4f14-41ba-b486-e3f7e44288d4" containerID="d80f1865979edfa596f07a94d96385edb66cffd70afa17a2d35a1f5543032ef0" exitCode=0 Oct 30 18:47:32.730067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.729959 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" Oct 30 18:47:32.730067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.729962 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" event={"ID":"20bb7d71-4f14-41ba-b486-e3f7e44288d4","Type":"ContainerDied","Data":"d80f1865979edfa596f07a94d96385edb66cffd70afa17a2d35a1f5543032ef0"} Oct 30 18:47:32.730151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.730084 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5" event={"ID":"20bb7d71-4f14-41ba-b486-e3f7e44288d4","Type":"ContainerDied","Data":"13c70fdbec07f28e61573384a1829379d03bad8672319089a861551318b08bc4"} Oct 30 18:47:32.730151 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.730107 5232 scope.go:122] "RemoveContainer" containerID="d80f1865979edfa596f07a94d96385edb66cffd70afa17a2d35a1f5543032ef0" Oct 30 18:47:32.739022 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.739004 5232 scope.go:122] "RemoveContainer" containerID="d80f1865979edfa596f07a94d96385edb66cffd70afa17a2d35a1f5543032ef0" Oct 30 18:47:32.739282 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:47:32.739260 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d80f1865979edfa596f07a94d96385edb66cffd70afa17a2d35a1f5543032ef0\": container with ID starting with d80f1865979edfa596f07a94d96385edb66cffd70afa17a2d35a1f5543032ef0 not found: ID does not exist" containerID="d80f1865979edfa596f07a94d96385edb66cffd70afa17a2d35a1f5543032ef0" Oct 30 18:47:32.739370 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.739295 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d80f1865979edfa596f07a94d96385edb66cffd70afa17a2d35a1f5543032ef0"} err="failed to get container status \"d80f1865979edfa596f07a94d96385edb66cffd70afa17a2d35a1f5543032ef0\": rpc error: code = NotFound desc = could not find container \"d80f1865979edfa596f07a94d96385edb66cffd70afa17a2d35a1f5543032ef0\": container with ID starting with d80f1865979edfa596f07a94d96385edb66cffd70afa17a2d35a1f5543032ef0 not found: ID does not exist" Oct 30 18:47:32.776621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.776569 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" podStartSLOduration=1.776556957 podStartE2EDuration="1.776556957s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:32.774608755 +0000 UTC m=+2752.091029074" watchObservedRunningTime="2026-10-30 18:47:32.776556957 +0000 UTC m=+2752.092977278" Oct 30 18:47:32.811942 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.811913 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5"] Oct 30 18:47:32.816605 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:32.816578 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-78b5489d4d-rqcd5"] Oct 30 18:47:33.531726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:33.531692 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:33.537078 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:33.537055 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:33.735569 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:33.735536 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:33.736636 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:33.736613 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:34.800411 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:34.800375 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20bb7d71-4f14-41ba-b486-e3f7e44288d4" path="/var/lib/kubelet/pods/20bb7d71-4f14-41ba-b486-e3f7e44288d4/volumes" Oct 30 18:47:37.357372 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.357339 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn"] Oct 30 18:47:37.357918 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.357896 5232 cpu_manager.go:414] "containerMap: removing container" podUID="20bb7d71-4f14-41ba-b486-e3f7e44288d4" containerName="maas-api" Oct 30 18:47:37.357991 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.357916 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20bb7d71-4f14-41ba-b486-e3f7e44288d4" containerName="maas-api" Oct 30 18:47:37.358069 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.358054 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="20bb7d71-4f14-41ba-b486-e3f7e44288d4" containerName="maas-api" Oct 30 18:47:37.362290 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.362268 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" Oct 30 18:47:37.381106 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.381085 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-069c47a6-serving-cert\"" Oct 30 18:47:37.391561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.391530 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn"] Oct 30 18:47:37.429407 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.429380 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6bcwb\" (UniqueName: \"kubernetes.io/projected/7566da62-2789-4a55-9fcb-1db2080dc695-kube-api-access-6bcwb\") pod \"maas-api-e2e-ait-069c47a6-57fb749-qk7xn\" (UID: \"7566da62-2789-4a55-9fcb-1db2080dc695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" Oct 30 18:47:37.429592 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.429435 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7566da62-2789-4a55-9fcb-1db2080dc695-metrics-certs\") pod \"maas-api-e2e-ait-069c47a6-57fb749-qk7xn\" (UID: \"7566da62-2789-4a55-9fcb-1db2080dc695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" Oct 30 18:47:37.429637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.429588 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7566da62-2789-4a55-9fcb-1db2080dc695-maas-api-tls\") pod \"maas-api-e2e-ait-069c47a6-57fb749-qk7xn\" (UID: \"7566da62-2789-4a55-9fcb-1db2080dc695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" Oct 30 18:47:37.429680 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.429651 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7566da62-2789-4a55-9fcb-1db2080dc695-openshift-service-ca\") pod \"maas-api-e2e-ait-069c47a6-57fb749-qk7xn\" (UID: \"7566da62-2789-4a55-9fcb-1db2080dc695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" Oct 30 18:47:37.531083 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.531048 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7566da62-2789-4a55-9fcb-1db2080dc695-metrics-certs\") pod \"maas-api-e2e-ait-069c47a6-57fb749-qk7xn\" (UID: \"7566da62-2789-4a55-9fcb-1db2080dc695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" Oct 30 18:47:37.531290 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.531104 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7566da62-2789-4a55-9fcb-1db2080dc695-maas-api-tls\") pod \"maas-api-e2e-ait-069c47a6-57fb749-qk7xn\" (UID: \"7566da62-2789-4a55-9fcb-1db2080dc695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" Oct 30 18:47:37.531290 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.531163 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7566da62-2789-4a55-9fcb-1db2080dc695-openshift-service-ca\") pod \"maas-api-e2e-ait-069c47a6-57fb749-qk7xn\" (UID: \"7566da62-2789-4a55-9fcb-1db2080dc695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" Oct 30 18:47:37.531290 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.531250 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6bcwb\" (UniqueName: \"kubernetes.io/projected/7566da62-2789-4a55-9fcb-1db2080dc695-kube-api-access-6bcwb\") pod \"maas-api-e2e-ait-069c47a6-57fb749-qk7xn\" (UID: \"7566da62-2789-4a55-9fcb-1db2080dc695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" Oct 30 18:47:37.531909 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.531834 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7566da62-2789-4a55-9fcb-1db2080dc695-openshift-service-ca\") pod \"maas-api-e2e-ait-069c47a6-57fb749-qk7xn\" (UID: \"7566da62-2789-4a55-9fcb-1db2080dc695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" Oct 30 18:47:37.533738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.533710 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7566da62-2789-4a55-9fcb-1db2080dc695-metrics-certs\") pod \"maas-api-e2e-ait-069c47a6-57fb749-qk7xn\" (UID: \"7566da62-2789-4a55-9fcb-1db2080dc695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" Oct 30 18:47:37.533881 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.533812 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7566da62-2789-4a55-9fcb-1db2080dc695-maas-api-tls\") pod \"maas-api-e2e-ait-069c47a6-57fb749-qk7xn\" (UID: \"7566da62-2789-4a55-9fcb-1db2080dc695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" Oct 30 18:47:37.551561 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.551530 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6bcwb\" (UniqueName: \"kubernetes.io/projected/7566da62-2789-4a55-9fcb-1db2080dc695-kube-api-access-6bcwb\") pod \"maas-api-e2e-ait-069c47a6-57fb749-qk7xn\" (UID: \"7566da62-2789-4a55-9fcb-1db2080dc695\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" Oct 30 18:47:37.679726 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.679650 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" Oct 30 18:47:37.712533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.712482 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8"] Oct 30 18:47:37.716075 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.716053 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8" Oct 30 18:47:37.736544 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.735986 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-069c47a6-dockercfg-2gnng\"" Oct 30 18:47:37.736700 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.736676 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-069c47a6\"" Oct 30 18:47:37.741207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.741129 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8"] Oct 30 18:47:37.802239 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.802206 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd"] Oct 30 18:47:37.806854 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.806826 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd" Oct 30 18:47:37.827562 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.827525 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd"] Oct 30 18:47:37.834092 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.834067 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vwcln\" (UniqueName: \"kubernetes.io/projected/47bb8bec-f97b-45bf-b0d4-179d016c7af9-kube-api-access-vwcln\") pod \"payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8\" (UID: \"47bb8bec-f97b-45bf-b0d4-179d016c7af9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8" Oct 30 18:47:37.834320 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.834248 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bb8bec-f97b-45bf-b0d4-179d016c7af9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8\" (UID: \"47bb8bec-f97b-45bf-b0d4-179d016c7af9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8" Oct 30 18:47:37.935273 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.935198 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47cf1331-4ce8-4731-ba03-3bf4f4dc79e8-plugins-config-volume\") pod \"payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd\" (UID: \"47cf1331-4ce8-4731-ba03-3bf4f4dc79e8\") " pod="openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd" Oct 30 18:47:37.935273 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.935248 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bb8bec-f97b-45bf-b0d4-179d016c7af9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8\" (UID: \"47bb8bec-f97b-45bf-b0d4-179d016c7af9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8" Oct 30 18:47:37.935489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.935313 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q5lk4\" (UniqueName: \"kubernetes.io/projected/47cf1331-4ce8-4731-ba03-3bf4f4dc79e8-kube-api-access-q5lk4\") pod \"payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd\" (UID: \"47cf1331-4ce8-4731-ba03-3bf4f4dc79e8\") " pod="openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd" Oct 30 18:47:37.935489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.935357 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vwcln\" (UniqueName: \"kubernetes.io/projected/47bb8bec-f97b-45bf-b0d4-179d016c7af9-kube-api-access-vwcln\") pod \"payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8\" (UID: \"47bb8bec-f97b-45bf-b0d4-179d016c7af9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8" Oct 30 18:47:37.935997 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.935975 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bb8bec-f97b-45bf-b0d4-179d016c7af9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8\" (UID: \"47bb8bec-f97b-45bf-b0d4-179d016c7af9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8" Oct 30 18:47:37.957632 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.957610 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vwcln\" (UniqueName: \"kubernetes.io/projected/47bb8bec-f97b-45bf-b0d4-179d016c7af9-kube-api-access-vwcln\") pod \"payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8\" (UID: \"47bb8bec-f97b-45bf-b0d4-179d016c7af9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8" Oct 30 18:47:37.974492 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:37.974432 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn"] Oct 30 18:47:37.981361 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:47:37.981321 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7566da62_2789_4a55_9fcb_1db2080dc695.slice/crio-91887e15a044188f8e4ef3aaaf8643bcb461a52469303e2558c65585d1caac2d WatchSource:0}: Error finding container 91887e15a044188f8e4ef3aaaf8643bcb461a52469303e2558c65585d1caac2d: Status 404 returned error can't find the container with id 91887e15a044188f8e4ef3aaaf8643bcb461a52469303e2558c65585d1caac2d Oct 30 18:47:38.029689 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.029661 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8" Oct 30 18:47:38.036837 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.036816 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47cf1331-4ce8-4731-ba03-3bf4f4dc79e8-plugins-config-volume\") pod \"payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd\" (UID: \"47cf1331-4ce8-4731-ba03-3bf4f4dc79e8\") " pod="openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd" Oct 30 18:47:38.036898 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.036877 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q5lk4\" (UniqueName: \"kubernetes.io/projected/47cf1331-4ce8-4731-ba03-3bf4f4dc79e8-kube-api-access-q5lk4\") pod \"payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd\" (UID: \"47cf1331-4ce8-4731-ba03-3bf4f4dc79e8\") " pod="openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd" Oct 30 18:47:38.037463 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.037432 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47cf1331-4ce8-4731-ba03-3bf4f4dc79e8-plugins-config-volume\") pod \"payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd\" (UID: \"47cf1331-4ce8-4731-ba03-3bf4f4dc79e8\") " pod="openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd" Oct 30 18:47:38.056568 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.056542 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q5lk4\" (UniqueName: \"kubernetes.io/projected/47cf1331-4ce8-4731-ba03-3bf4f4dc79e8-kube-api-access-q5lk4\") pod \"payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd\" (UID: \"47cf1331-4ce8-4731-ba03-3bf4f4dc79e8\") " pod="openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd" Oct 30 18:47:38.122378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.122350 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd" Oct 30 18:47:38.318864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.318752 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8"] Oct 30 18:47:38.320935 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:47:38.320907 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47bb8bec_f97b_45bf_b0d4_179d016c7af9.slice/crio-cdb5b8280b94afa4398fee0971f3d03eaa3dfadf3419eab99d8ed630cfc01cf7 WatchSource:0}: Error finding container cdb5b8280b94afa4398fee0971f3d03eaa3dfadf3419eab99d8ed630cfc01cf7: Status 404 returned error can't find the container with id cdb5b8280b94afa4398fee0971f3d03eaa3dfadf3419eab99d8ed630cfc01cf7 Oct 30 18:47:38.421893 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.417902 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd"] Oct 30 18:47:38.764229 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.764190 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" event={"ID":"7566da62-2789-4a55-9fcb-1db2080dc695","Type":"ContainerStarted","Data":"6166066f09227757fb235bde0a869523f09d6a4024ed950b2ea64b7eb781795e"} Oct 30 18:47:38.764432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.764252 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" event={"ID":"7566da62-2789-4a55-9fcb-1db2080dc695","Type":"ContainerStarted","Data":"91887e15a044188f8e4ef3aaaf8643bcb461a52469303e2558c65585d1caac2d"} Oct 30 18:47:38.764432 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.764312 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" Oct 30 18:47:38.765928 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.765902 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd" event={"ID":"47cf1331-4ce8-4731-ba03-3bf4f4dc79e8","Type":"ContainerStarted","Data":"1bd54863b07948567ef680c6cfee97256cc514db1b01a435c3bccf7e37d57217"} Oct 30 18:47:38.766069 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.765948 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd" event={"ID":"47cf1331-4ce8-4731-ba03-3bf4f4dc79e8","Type":"ContainerStarted","Data":"9297b5a5314c53728b51dd144ab694405859c9d086f5df758ea7d2aa58a07f5d"} Oct 30 18:47:38.766069 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.765998 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd" Oct 30 18:47:38.767347 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.767321 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8" event={"ID":"47bb8bec-f97b-45bf-b0d4-179d016c7af9","Type":"ContainerStarted","Data":"2accf30b97ceb3dbfaf0eeecb5931fe6286f09d72799fba99d4d3ddec3984132"} Oct 30 18:47:38.767449 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.767353 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8" event={"ID":"47bb8bec-f97b-45bf-b0d4-179d016c7af9","Type":"ContainerStarted","Data":"cdb5b8280b94afa4398fee0971f3d03eaa3dfadf3419eab99d8ed630cfc01cf7"} Oct 30 18:47:38.767449 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.767372 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8" Oct 30 18:47:38.805826 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.805781 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" podStartSLOduration=1.439848945 podStartE2EDuration="1.805768881s" totalImagesPullingTime="365.919936ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:38.805483204 +0000 UTC m=+2758.121903525" watchObservedRunningTime="2026-10-30 18:47:38.805768881 +0000 UTC m=+2758.122189221" Oct 30 18:47:38.844627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.844560 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd" podStartSLOduration=1.844542584 podStartE2EDuration="1.844542584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:38.842619419 +0000 UTC m=+2758.159039737" watchObservedRunningTime="2026-10-30 18:47:38.844542584 +0000 UTC m=+2758.160962907" Oct 30 18:47:38.882716 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:38.882672 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8" podStartSLOduration=1.882656972 podStartE2EDuration="1.882656972s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:47:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:47:38.88188924 +0000 UTC m=+2758.198309562" watchObservedRunningTime="2026-10-30 18:47:38.882656972 +0000 UTC m=+2758.199077330" Oct 30 18:47:44.778080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:44.778053 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" Oct 30 18:47:49.775994 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:49.775952 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8" Oct 30 18:47:49.776529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:49.776040 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd" Oct 30 18:47:49.778648 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:49.778614 5232 patch_prober.go:29] interesting pod/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 30 18:47:49.778755 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:49.778687 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" podUID="d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 30 18:47:50.829809 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:50.829771 5232 generic.go:356] "Generic (PLEG): container finished" podID="d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" containerID="98bc08a7ab3ca8c3871fccdb5f691ccb5b71ca0377f5d0db8840f098e4989c3f" exitCode=0 Oct 30 18:47:50.830235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:50.829840 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" event={"ID":"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d","Type":"ContainerDied","Data":"98bc08a7ab3ca8c3871fccdb5f691ccb5b71ca0377f5d0db8840f098e4989c3f"} Oct 30 18:47:50.830235 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:50.830148 5232 scope.go:122] "RemoveContainer" containerID="98bc08a7ab3ca8c3871fccdb5f691ccb5b71ca0377f5d0db8840f098e4989c3f" Oct 30 18:47:50.831292 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:50.831265 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:47:50.831352 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:50.831339 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:47:50.831391 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:50.831372 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:47:51.841480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:51.841446 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" event={"ID":"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d","Type":"ContainerStarted","Data":"90a1c0be098f913401fc059e7d0d0ef87166556becab4be8f686ccc12e63cc0e"} Oct 30 18:47:52.534682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:52.534653 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:52.535051 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:52.535018 5232 patch_prober.go:29] interesting pod/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body= Oct 30 18:47:52.535209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:52.535100 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" podUID="d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" Oct 30 18:47:53.534574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:53.532835 5232 patch_prober.go:29] interesting pod/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body= Oct 30 18:47:53.534574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:53.532892 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" podUID="d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" Oct 30 18:47:54.533535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:54.532911 5232 patch_prober.go:29] interesting pod/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body= Oct 30 18:47:54.533535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:54.532973 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" podUID="d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" Oct 30 18:47:55.534673 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:55.534636 5232 patch_prober.go:29] interesting pod/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body= Oct 30 18:47:55.535339 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:55.534695 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" podUID="d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" Oct 30 18:47:56.536533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:56.533612 5232 patch_prober.go:29] interesting pod/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" start-of-body= Oct 30 18:47:56.536533 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:56.533691 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" podUID="d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.29:15021/healthz/ready\": dial tcp 10.133.1.29:15021: connect: connection refused" Oct 30 18:47:57.031925 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:47:57.031878 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:47:57.535962 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:57.535934 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:57.536306 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:57.536288 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:47:57.537031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:47:57.537006 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:48:00.746125 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:00.746095 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd"] Oct 30 18:48:00.778852 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:00.778817 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8"] Oct 30 18:48:02.640571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:02.640538 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn"] Oct 30 18:48:02.641091 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:02.640721 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" podUID="7566da62-2789-4a55-9fcb-1db2080dc695" containerName="maas-api" containerID="cri-o://6166066f09227757fb235bde0a869523f09d6a4024ed950b2ea64b7eb781795e" gracePeriod=30 Oct 30 18:48:02.900558 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:02.900449 5232 generic.go:356] "Generic (PLEG): container finished" podID="7566da62-2789-4a55-9fcb-1db2080dc695" containerID="6166066f09227757fb235bde0a869523f09d6a4024ed950b2ea64b7eb781795e" exitCode=0 Oct 30 18:48:02.907729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:02.900584 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" event={"ID":"7566da62-2789-4a55-9fcb-1db2080dc695","Type":"ContainerDied","Data":"6166066f09227757fb235bde0a869523f09d6a4024ed950b2ea64b7eb781795e"} Oct 30 18:48:02.907729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:02.900665 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" event={"ID":"7566da62-2789-4a55-9fcb-1db2080dc695","Type":"ContainerDied","Data":"91887e15a044188f8e4ef3aaaf8643bcb461a52469303e2558c65585d1caac2d"} Oct 30 18:48:02.907729 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:02.900689 5232 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="91887e15a044188f8e4ef3aaaf8643bcb461a52469303e2558c65585d1caac2d" Oct 30 18:48:02.909752 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:02.909727 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" Oct 30 18:48:02.919812 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:02.919788 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7566da62-2789-4a55-9fcb-1db2080dc695-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7566da62-2789-4a55-9fcb-1db2080dc695-maas-api-tls\") pod \"7566da62-2789-4a55-9fcb-1db2080dc695\" (UID: \"7566da62-2789-4a55-9fcb-1db2080dc695\") " Oct 30 18:48:02.919952 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:02.919830 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7566da62-2789-4a55-9fcb-1db2080dc695-kube-api-access-6bcwb\" (UniqueName: \"kubernetes.io/projected/7566da62-2789-4a55-9fcb-1db2080dc695-kube-api-access-6bcwb\") pod \"7566da62-2789-4a55-9fcb-1db2080dc695\" (UID: \"7566da62-2789-4a55-9fcb-1db2080dc695\") " Oct 30 18:48:02.919952 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:02.919884 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7566da62-2789-4a55-9fcb-1db2080dc695-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7566da62-2789-4a55-9fcb-1db2080dc695-metrics-certs\") pod \"7566da62-2789-4a55-9fcb-1db2080dc695\" (UID: \"7566da62-2789-4a55-9fcb-1db2080dc695\") " Oct 30 18:48:02.919952 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:02.919911 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7566da62-2789-4a55-9fcb-1db2080dc695-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7566da62-2789-4a55-9fcb-1db2080dc695-openshift-service-ca\") pod \"7566da62-2789-4a55-9fcb-1db2080dc695\" (UID: \"7566da62-2789-4a55-9fcb-1db2080dc695\") " Oct 30 18:48:02.920329 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:02.920304 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7566da62-2789-4a55-9fcb-1db2080dc695-openshift-service-ca" pod "7566da62-2789-4a55-9fcb-1db2080dc695" (UID: "7566da62-2789-4a55-9fcb-1db2080dc695"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:02.922011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:02.921987 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7566da62-2789-4a55-9fcb-1db2080dc695-maas-api-tls" pod "7566da62-2789-4a55-9fcb-1db2080dc695" (UID: "7566da62-2789-4a55-9fcb-1db2080dc695"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:02.922105 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:02.922093 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7566da62-2789-4a55-9fcb-1db2080dc695-metrics-certs" pod "7566da62-2789-4a55-9fcb-1db2080dc695" (UID: "7566da62-2789-4a55-9fcb-1db2080dc695"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:48:02.922172 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:02.922121 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7566da62-2789-4a55-9fcb-1db2080dc695-kube-api-access-6bcwb" pod "7566da62-2789-4a55-9fcb-1db2080dc695" (UID: "7566da62-2789-4a55-9fcb-1db2080dc695"). InnerVolumeSpecName "kube-api-access-6bcwb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:03.021450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:03.021413 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7566da62-2789-4a55-9fcb-1db2080dc695-maas-api-tls\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:48:03.021450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:03.021449 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6bcwb\" (UniqueName: \"kubernetes.io/projected/7566da62-2789-4a55-9fcb-1db2080dc695-kube-api-access-6bcwb\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:48:03.021697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:03.021462 5232 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7566da62-2789-4a55-9fcb-1db2080dc695-metrics-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:48:03.021697 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:03.021474 5232 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7566da62-2789-4a55-9fcb-1db2080dc695-openshift-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:48:03.907064 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:03.907032 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn" Oct 30 18:48:03.957230 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:03.957203 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn"] Oct 30 18:48:03.969601 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:03.969576 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-069c47a6-57fb749-qk7xn"] Oct 30 18:48:04.800864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:04.800834 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7566da62-2789-4a55-9fcb-1db2080dc695" path="/var/lib/kubelet/pods/7566da62-2789-4a55-9fcb-1db2080dc695/volumes" Oct 30 18:48:05.748063 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:05.747048 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd" podUID="47cf1331-4ce8-4731-ba03-3bf4f4dc79e8" containerName="payload-processing" containerID="cri-o://1bd54863b07948567ef680c6cfee97256cc514db1b01a435c3bccf7e37d57217" gracePeriod=25 Oct 30 18:48:05.779678 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:05.779626 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8" podUID="47bb8bec-f97b-45bf-b0d4-179d016c7af9" containerName="payload-pre-processing" containerID="cri-o://2accf30b97ceb3dbfaf0eeecb5931fe6286f09d72799fba99d4d3ddec3984132" gracePeriod=25 Oct 30 18:48:05.919200 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:05.919167 5232 generic.go:356] "Generic (PLEG): container finished" podID="47cf1331-4ce8-4731-ba03-3bf4f4dc79e8" containerID="1bd54863b07948567ef680c6cfee97256cc514db1b01a435c3bccf7e37d57217" exitCode=0 Oct 30 18:48:05.919374 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:05.919242 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd" event={"ID":"47cf1331-4ce8-4731-ba03-3bf4f4dc79e8","Type":"ContainerDied","Data":"1bd54863b07948567ef680c6cfee97256cc514db1b01a435c3bccf7e37d57217"} Oct 30 18:48:05.920939 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:05.920914 5232 generic.go:356] "Generic (PLEG): container finished" podID="47bb8bec-f97b-45bf-b0d4-179d016c7af9" containerID="2accf30b97ceb3dbfaf0eeecb5931fe6286f09d72799fba99d4d3ddec3984132" exitCode=0 Oct 30 18:48:05.921099 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:05.920975 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8" event={"ID":"47bb8bec-f97b-45bf-b0d4-179d016c7af9","Type":"ContainerDied","Data":"2accf30b97ceb3dbfaf0eeecb5931fe6286f09d72799fba99d4d3ddec3984132"} Oct 30 18:48:06.020343 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.020319 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd" Oct 30 18:48:06.047790 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.047766 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47cf1331-4ce8-4731-ba03-3bf4f4dc79e8-kube-api-access-q5lk4\" (UniqueName: \"kubernetes.io/projected/47cf1331-4ce8-4731-ba03-3bf4f4dc79e8-kube-api-access-q5lk4\") pod \"47cf1331-4ce8-4731-ba03-3bf4f4dc79e8\" (UID: \"47cf1331-4ce8-4731-ba03-3bf4f4dc79e8\") " Oct 30 18:48:06.047928 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.047806 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47cf1331-4ce8-4731-ba03-3bf4f4dc79e8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47cf1331-4ce8-4731-ba03-3bf4f4dc79e8-plugins-config-volume\") pod \"47cf1331-4ce8-4731-ba03-3bf4f4dc79e8\" (UID: \"47cf1331-4ce8-4731-ba03-3bf4f4dc79e8\") " Oct 30 18:48:06.048258 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.048233 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47cf1331-4ce8-4731-ba03-3bf4f4dc79e8-plugins-config-volume" pod "47cf1331-4ce8-4731-ba03-3bf4f4dc79e8" (UID: "47cf1331-4ce8-4731-ba03-3bf4f4dc79e8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:06.049966 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.049940 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47cf1331-4ce8-4731-ba03-3bf4f4dc79e8-kube-api-access-q5lk4" pod "47cf1331-4ce8-4731-ba03-3bf4f4dc79e8" (UID: "47cf1331-4ce8-4731-ba03-3bf4f4dc79e8"). InnerVolumeSpecName "kube-api-access-q5lk4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:06.051112 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.051091 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8" Oct 30 18:48:06.148457 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.148429 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47bb8bec-f97b-45bf-b0d4-179d016c7af9-kube-api-access-vwcln\" (UniqueName: \"kubernetes.io/projected/47bb8bec-f97b-45bf-b0d4-179d016c7af9-kube-api-access-vwcln\") pod \"47bb8bec-f97b-45bf-b0d4-179d016c7af9\" (UID: \"47bb8bec-f97b-45bf-b0d4-179d016c7af9\") " Oct 30 18:48:06.148656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.148538 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47bb8bec-f97b-45bf-b0d4-179d016c7af9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bb8bec-f97b-45bf-b0d4-179d016c7af9-plugins-config-volume\") pod \"47bb8bec-f97b-45bf-b0d4-179d016c7af9\" (UID: \"47bb8bec-f97b-45bf-b0d4-179d016c7af9\") " Oct 30 18:48:06.148845 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.148824 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q5lk4\" (UniqueName: \"kubernetes.io/projected/47cf1331-4ce8-4731-ba03-3bf4f4dc79e8-kube-api-access-q5lk4\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:48:06.148891 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.148852 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47cf1331-4ce8-4731-ba03-3bf4f4dc79e8-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:48:06.148891 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.148864 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47bb8bec-f97b-45bf-b0d4-179d016c7af9-plugins-config-volume" pod "47bb8bec-f97b-45bf-b0d4-179d016c7af9" (UID: "47bb8bec-f97b-45bf-b0d4-179d016c7af9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:06.150759 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.150728 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47bb8bec-f97b-45bf-b0d4-179d016c7af9-kube-api-access-vwcln" pod "47bb8bec-f97b-45bf-b0d4-179d016c7af9" (UID: "47bb8bec-f97b-45bf-b0d4-179d016c7af9"). InnerVolumeSpecName "kube-api-access-vwcln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:06.250223 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.250153 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vwcln\" (UniqueName: \"kubernetes.io/projected/47bb8bec-f97b-45bf-b0d4-179d016c7af9-kube-api-access-vwcln\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:48:06.250223 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.250178 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/47bb8bec-f97b-45bf-b0d4-179d016c7af9-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:48:06.493033 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.493000 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl"] Oct 30 18:48:06.493264 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.493240 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" podUID="d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" containerName="istio-proxy" containerID="cri-o://90a1c0be098f913401fc059e7d0d0ef87166556becab4be8f686ccc12e63cc0e" gracePeriod=30 Oct 30 18:48:06.926979 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.926950 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd" Oct 30 18:48:06.927400 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.926948 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd" event={"ID":"47cf1331-4ce8-4731-ba03-3bf4f4dc79e8","Type":"ContainerDied","Data":"9297b5a5314c53728b51dd144ab694405859c9d086f5df758ea7d2aa58a07f5d"} Oct 30 18:48:06.927400 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.927095 5232 scope.go:122] "RemoveContainer" containerID="1bd54863b07948567ef680c6cfee97256cc514db1b01a435c3bccf7e37d57217" Oct 30 18:48:06.928463 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.928426 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8" event={"ID":"47bb8bec-f97b-45bf-b0d4-179d016c7af9","Type":"ContainerDied","Data":"cdb5b8280b94afa4398fee0971f3d03eaa3dfadf3419eab99d8ed630cfc01cf7"} Oct 30 18:48:06.928591 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.928467 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8" Oct 30 18:48:06.936231 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.936215 5232 scope.go:122] "RemoveContainer" containerID="2accf30b97ceb3dbfaf0eeecb5931fe6286f09d72799fba99d4d3ddec3984132" Oct 30 18:48:06.977399 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.977379 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd"] Oct 30 18:48:06.984079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:06.984059 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-069c47a6-96db766f8-7jhqd"] Oct 30 18:48:07.017687 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:07.017640 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8"] Oct 30 18:48:07.025031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:07.025009 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-069c47a6-6d64bfdb59-sx2j8"] Oct 30 18:48:08.251195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.251161 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph"] Oct 30 18:48:08.251780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.251543 5232 cpu_manager.go:414] "containerMap: removing container" podUID="47bb8bec-f97b-45bf-b0d4-179d016c7af9" containerName="payload-pre-processing" Oct 30 18:48:08.251780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.251556 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47bb8bec-f97b-45bf-b0d4-179d016c7af9" containerName="payload-pre-processing" Oct 30 18:48:08.251780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.251572 5232 cpu_manager.go:414] "containerMap: removing container" podUID="7566da62-2789-4a55-9fcb-1db2080dc695" containerName="maas-api" Oct 30 18:48:08.251780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.251577 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7566da62-2789-4a55-9fcb-1db2080dc695" containerName="maas-api" Oct 30 18:48:08.251780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.251601 5232 cpu_manager.go:414] "containerMap: removing container" podUID="47cf1331-4ce8-4731-ba03-3bf4f4dc79e8" containerName="payload-processing" Oct 30 18:48:08.251780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.251606 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47cf1331-4ce8-4731-ba03-3bf4f4dc79e8" containerName="payload-processing" Oct 30 18:48:08.251780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.251671 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="47cf1331-4ce8-4731-ba03-3bf4f4dc79e8" containerName="payload-processing" Oct 30 18:48:08.251780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.251684 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="47bb8bec-f97b-45bf-b0d4-179d016c7af9" containerName="payload-pre-processing" Oct 30 18:48:08.251780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.251692 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="7566da62-2789-4a55-9fcb-1db2080dc695" containerName="maas-api" Oct 30 18:48:08.254675 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.254657 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.269318 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.269287 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-envoy\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.269456 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.269332 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sl82r\" (UniqueName: \"kubernetes.io/projected/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-kube-api-access-sl82r\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.269456 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.269391 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-credential-socket\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.269456 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.269426 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-workload-socket\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.269456 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.269443 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istiod-ca-cert\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.269746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.269468 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-workload-certs\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.269746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.269643 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-data\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.269746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.269677 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-token\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.269916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.269746 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-podinfo\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.270769 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.270750 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-783444-openshift-default-dockercfg-htcg4\"" Oct 30 18:48:08.284682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.284654 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph"] Oct 30 18:48:08.370303 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.370269 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-workload-socket\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.370303 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.370307 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istiod-ca-cert\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.370574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.370333 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-workload-certs\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.370574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.370359 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-data\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.370574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.370394 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-token\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.370574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.370446 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-podinfo\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.370574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.370476 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-envoy\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.370574 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.370551 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sl82r\" (UniqueName: \"kubernetes.io/projected/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-kube-api-access-sl82r\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.370924 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.370594 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-credential-socket\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.370924 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.370719 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-workload-socket\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.371082 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.371064 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-workload-certs\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.371164 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.371141 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-data\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.371254 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.371230 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-credential-socket\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.371606 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.371583 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istiod-ca-cert\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.373078 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.373055 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-envoy\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.373184 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.373159 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-podinfo\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.389896 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.389868 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-token\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.390022 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.389970 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sl82r\" (UniqueName: \"kubernetes.io/projected/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-kube-api-access-sl82r\") pod \"e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.566992 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.566910 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:08.800618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.800578 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47bb8bec-f97b-45bf-b0d4-179d016c7af9" path="/var/lib/kubelet/pods/47bb8bec-f97b-45bf-b0d4-179d016c7af9/volumes" Oct 30 18:48:08.801031 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.801010 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47cf1331-4ce8-4731-ba03-3bf4f4dc79e8" path="/var/lib/kubelet/pods/47cf1331-4ce8-4731-ba03-3bf4f4dc79e8/volumes" Oct 30 18:48:08.890296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.890270 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph"] Oct 30 18:48:08.892409 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:48:08.892379 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6d0a2ef2_20b4_4c2f_ae84_119dec8c235f.slice/crio-a52bc5b3de8e5017951c85e04410b4066bcc7d57f3a01ecad95b170b10a6db14 WatchSource:0}: Error finding container a52bc5b3de8e5017951c85e04410b4066bcc7d57f3a01ecad95b170b10a6db14: Status 404 returned error can't find the container with id a52bc5b3de8e5017951c85e04410b4066bcc7d57f3a01ecad95b170b10a6db14 Oct 30 18:48:08.894500 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.894465 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:48:08.894639 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.894565 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:48:08.894639 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.894609 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:48:08.941467 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:08.941438 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" event={"ID":"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f","Type":"ContainerStarted","Data":"a52bc5b3de8e5017951c85e04410b4066bcc7d57f3a01ecad95b170b10a6db14"} Oct 30 18:48:09.948041 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:09.948009 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" event={"ID":"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f","Type":"ContainerStarted","Data":"9094ddab6d3f6d483ea318590664af3c019bc230c6fcb22406c07f81cbb91ee2"} Oct 30 18:48:10.010274 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:10.010218 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" podStartSLOduration=2.010204988 podStartE2EDuration="2.010204988s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:10.009104169 +0000 UTC m=+2789.325524489" watchObservedRunningTime="2026-10-30 18:48:10.010204988 +0000 UTC m=+2789.326625381" Oct 30 18:48:10.567685 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:10.567652 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:10.572426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:10.572404 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:10.952365 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:10.952287 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:10.953404 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:10.953386 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:11.746918 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.746895 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:48:11.804299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.804219 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-credential-socket\") pod \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " Oct 30 18:48:11.804299 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.804282 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-token\" (UniqueName: \"kubernetes.io/projected/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-token\") pod \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " Oct 30 18:48:11.804627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.804319 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-data\") pod \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " Oct 30 18:48:11.804627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.804399 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-workload-socket\") pod \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " Oct 30 18:48:11.804627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.804450 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-podinfo\") pod \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " Oct 30 18:48:11.804627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.804481 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istiod-ca-cert\") pod \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " Oct 30 18:48:11.804627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.804597 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-envoy\") pod \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " Oct 30 18:48:11.804627 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.804624 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-workload-certs\") pod \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " Oct 30 18:48:11.804978 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.804660 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-kube-api-access-k6624\" (UniqueName: \"kubernetes.io/projected/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-kube-api-access-k6624\") pod \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\" (UID: \"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d\") " Oct 30 18:48:11.805486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.804618 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-credential-socket" pod "d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" (UID: "d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:48:11.805637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.804854 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-workload-socket" pod "d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" (UID: "d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:48:11.805637 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.805248 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-data" pod "d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" (UID: "d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:48:11.805792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.805693 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-workload-certs" pod "d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" (UID: "d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:48:11.806215 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.806189 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istiod-ca-cert" pod "d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" (UID: "d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:48:11.807781 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.807743 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-token" pod "d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" (UID: "d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:11.808044 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.808004 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-podinfo" pod "d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" (UID: "d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:48:11.808291 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.808266 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-kube-api-access-k6624" pod "d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" (UID: "d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d"). InnerVolumeSpecName "kube-api-access-k6624". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:48:11.809149 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.809114 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-envoy" pod "d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" (UID: "d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:48:11.905378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.905346 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:48:11.905378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.905375 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:48:11.905378 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.905384 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:48:11.905644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.905393 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:48:11.905644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.905402 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:48:11.905644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.905411 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k6624\" (UniqueName: \"kubernetes.io/projected/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-kube-api-access-k6624\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:48:11.905644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.905421 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:48:11.905644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.905431 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:48:11.905644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.905439 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:48:11.958313 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.958275 5232 generic.go:356] "Generic (PLEG): container finished" podID="d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" containerID="90a1c0be098f913401fc059e7d0d0ef87166556becab4be8f686ccc12e63cc0e" exitCode=0 Oct 30 18:48:11.958818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.958335 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" Oct 30 18:48:11.958818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.958367 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" event={"ID":"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d","Type":"ContainerDied","Data":"90a1c0be098f913401fc059e7d0d0ef87166556becab4be8f686ccc12e63cc0e"} Oct 30 18:48:11.958818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.958422 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl" event={"ID":"d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d","Type":"ContainerDied","Data":"2b54b3ee07c873f7b57876c297b400e062e6bb924c266abe70b2edd6244634d0"} Oct 30 18:48:11.958818 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.958452 5232 scope.go:122] "RemoveContainer" containerID="90a1c0be098f913401fc059e7d0d0ef87166556becab4be8f686ccc12e63cc0e" Oct 30 18:48:11.968042 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.968028 5232 scope.go:122] "RemoveContainer" containerID="98bc08a7ab3ca8c3871fccdb5f691ccb5b71ca0377f5d0db8840f098e4989c3f" Oct 30 18:48:11.976023 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.976007 5232 scope.go:122] "RemoveContainer" containerID="90a1c0be098f913401fc059e7d0d0ef87166556becab4be8f686ccc12e63cc0e" Oct 30 18:48:11.976272 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:48:11.976255 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90a1c0be098f913401fc059e7d0d0ef87166556becab4be8f686ccc12e63cc0e\": container with ID starting with 90a1c0be098f913401fc059e7d0d0ef87166556becab4be8f686ccc12e63cc0e not found: ID does not exist" containerID="90a1c0be098f913401fc059e7d0d0ef87166556becab4be8f686ccc12e63cc0e" Oct 30 18:48:11.976323 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.976281 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90a1c0be098f913401fc059e7d0d0ef87166556becab4be8f686ccc12e63cc0e"} err="failed to get container status \"90a1c0be098f913401fc059e7d0d0ef87166556becab4be8f686ccc12e63cc0e\": rpc error: code = NotFound desc = could not find container \"90a1c0be098f913401fc059e7d0d0ef87166556becab4be8f686ccc12e63cc0e\": container with ID starting with 90a1c0be098f913401fc059e7d0d0ef87166556becab4be8f686ccc12e63cc0e not found: ID does not exist" Oct 30 18:48:11.976323 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.976296 5232 scope.go:122] "RemoveContainer" containerID="98bc08a7ab3ca8c3871fccdb5f691ccb5b71ca0377f5d0db8840f098e4989c3f" Oct 30 18:48:11.976521 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:48:11.976494 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"98bc08a7ab3ca8c3871fccdb5f691ccb5b71ca0377f5d0db8840f098e4989c3f\": container with ID starting with 98bc08a7ab3ca8c3871fccdb5f691ccb5b71ca0377f5d0db8840f098e4989c3f not found: ID does not exist" containerID="98bc08a7ab3ca8c3871fccdb5f691ccb5b71ca0377f5d0db8840f098e4989c3f" Oct 30 18:48:11.976571 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:11.976531 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"98bc08a7ab3ca8c3871fccdb5f691ccb5b71ca0377f5d0db8840f098e4989c3f"} err="failed to get container status \"98bc08a7ab3ca8c3871fccdb5f691ccb5b71ca0377f5d0db8840f098e4989c3f\": rpc error: code = NotFound desc = could not find container \"98bc08a7ab3ca8c3871fccdb5f691ccb5b71ca0377f5d0db8840f098e4989c3f\": container with ID starting with 98bc08a7ab3ca8c3871fccdb5f691ccb5b71ca0377f5d0db8840f098e4989c3f not found: ID does not exist" Oct 30 18:48:12.009127 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:12.009106 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl"] Oct 30 18:48:12.012622 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:12.012601 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-069c47a6-openshift-default-f9dcdf969-jfrhl"] Oct 30 18:48:12.801034 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:12.801005 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" path="/var/lib/kubelet/pods/d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d/volumes" Oct 30 18:48:13.712717 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.712684 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks"] Oct 30 18:48:13.713113 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.713034 5232 cpu_manager.go:414] "containerMap: removing container" podUID="d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" containerName="istio-proxy" Oct 30 18:48:13.713113 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.713046 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" containerName="istio-proxy" Oct 30 18:48:13.713113 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.713057 5232 cpu_manager.go:414] "containerMap: removing container" podUID="d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" containerName="istio-proxy" Oct 30 18:48:13.713113 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.713062 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" containerName="istio-proxy" Oct 30 18:48:13.713263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.713133 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" containerName="istio-proxy" Oct 30 18:48:13.713263 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.713142 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2dd47bb-3ad6-4e84-b4a9-a56c8655b98d" containerName="istio-proxy" Oct 30 18:48:13.718006 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.717988 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" Oct 30 18:48:13.745824 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.745793 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-783444-serving-cert\"" Oct 30 18:48:13.747463 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.747434 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks"] Oct 30 18:48:13.822792 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.822764 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfmrc\" (UniqueName: \"kubernetes.io/projected/6eaab134-7688-4e9d-8264-361d268cb91d-kube-api-access-kfmrc\") pod \"maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks\" (UID: \"6eaab134-7688-4e9d-8264-361d268cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" Oct 30 18:48:13.822963 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.822801 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eaab134-7688-4e9d-8264-361d268cb91d-metrics-certs\") pod \"maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks\" (UID: \"6eaab134-7688-4e9d-8264-361d268cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" Oct 30 18:48:13.822963 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.822953 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eaab134-7688-4e9d-8264-361d268cb91d-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks\" (UID: \"6eaab134-7688-4e9d-8264-361d268cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" Oct 30 18:48:13.823066 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.822983 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eaab134-7688-4e9d-8264-361d268cb91d-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks\" (UID: \"6eaab134-7688-4e9d-8264-361d268cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" Oct 30 18:48:13.924217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.924187 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfmrc\" (UniqueName: \"kubernetes.io/projected/6eaab134-7688-4e9d-8264-361d268cb91d-kube-api-access-kfmrc\") pod \"maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks\" (UID: \"6eaab134-7688-4e9d-8264-361d268cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" Oct 30 18:48:13.924375 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.924227 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eaab134-7688-4e9d-8264-361d268cb91d-metrics-certs\") pod \"maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks\" (UID: \"6eaab134-7688-4e9d-8264-361d268cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" Oct 30 18:48:13.924455 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.924436 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eaab134-7688-4e9d-8264-361d268cb91d-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks\" (UID: \"6eaab134-7688-4e9d-8264-361d268cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" Oct 30 18:48:13.924521 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.924472 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eaab134-7688-4e9d-8264-361d268cb91d-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks\" (UID: \"6eaab134-7688-4e9d-8264-361d268cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" Oct 30 18:48:13.925069 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.925043 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eaab134-7688-4e9d-8264-361d268cb91d-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks\" (UID: \"6eaab134-7688-4e9d-8264-361d268cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" Oct 30 18:48:13.926951 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.926930 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eaab134-7688-4e9d-8264-361d268cb91d-metrics-certs\") pod \"maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks\" (UID: \"6eaab134-7688-4e9d-8264-361d268cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" Oct 30 18:48:13.927063 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.926973 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eaab134-7688-4e9d-8264-361d268cb91d-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks\" (UID: \"6eaab134-7688-4e9d-8264-361d268cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" Oct 30 18:48:13.947740 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:13.947712 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfmrc\" (UniqueName: \"kubernetes.io/projected/6eaab134-7688-4e9d-8264-361d268cb91d-kube-api-access-kfmrc\") pod \"maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks\" (UID: \"6eaab134-7688-4e9d-8264-361d268cb91d\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" Oct 30 18:48:14.030215 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.030175 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" Oct 30 18:48:14.245296 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.245264 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf"] Oct 30 18:48:14.249864 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.249843 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf" Oct 30 18:48:14.267189 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.267157 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-783444\"" Oct 30 18:48:14.267345 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.267280 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-783444-dockercfg-krzh8\"" Oct 30 18:48:14.268499 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.268472 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf"] Oct 30 18:48:14.321559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.321450 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg"] Oct 30 18:48:14.325097 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.325063 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks"] Oct 30 18:48:14.325290 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.325176 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg" Oct 30 18:48:14.325825 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:48:14.325796 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6eaab134_7688_4e9d_8264_361d268cb91d.slice/crio-0265beee6c0fdffff07953067a4628258d0fe72b6f94188b0f7037b6932a4501 WatchSource:0}: Error finding container 0265beee6c0fdffff07953067a4628258d0fe72b6f94188b0f7037b6932a4501: Status 404 returned error can't find the container with id 0265beee6c0fdffff07953067a4628258d0fe72b6f94188b0f7037b6932a4501 Oct 30 18:48:14.328120 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.328088 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b309a532-1af6-4ec7-9f02-76cdeb71ae2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf\" (UID: \"b309a532-1af6-4ec7-9f02-76cdeb71ae2b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf" Oct 30 18:48:14.328283 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.328264 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkz7l\" (UniqueName: \"kubernetes.io/projected/b309a532-1af6-4ec7-9f02-76cdeb71ae2b-kube-api-access-zkz7l\") pod \"payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf\" (UID: \"b309a532-1af6-4ec7-9f02-76cdeb71ae2b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf" Oct 30 18:48:14.344039 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.344007 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg"] Oct 30 18:48:14.429389 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.429357 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fqqrj\" (UniqueName: \"kubernetes.io/projected/5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19-kube-api-access-fqqrj\") pod \"payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg\" (UID: \"5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg" Oct 30 18:48:14.429646 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.429406 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkz7l\" (UniqueName: \"kubernetes.io/projected/b309a532-1af6-4ec7-9f02-76cdeb71ae2b-kube-api-access-zkz7l\") pod \"payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf\" (UID: \"b309a532-1af6-4ec7-9f02-76cdeb71ae2b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf" Oct 30 18:48:14.429646 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.429536 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b309a532-1af6-4ec7-9f02-76cdeb71ae2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf\" (UID: \"b309a532-1af6-4ec7-9f02-76cdeb71ae2b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf" Oct 30 18:48:14.429646 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.429642 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg\" (UID: \"5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg" Oct 30 18:48:14.430209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.430186 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b309a532-1af6-4ec7-9f02-76cdeb71ae2b-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf\" (UID: \"b309a532-1af6-4ec7-9f02-76cdeb71ae2b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf" Oct 30 18:48:14.451683 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.451661 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkz7l\" (UniqueName: \"kubernetes.io/projected/b309a532-1af6-4ec7-9f02-76cdeb71ae2b-kube-api-access-zkz7l\") pod \"payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf\" (UID: \"b309a532-1af6-4ec7-9f02-76cdeb71ae2b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf" Oct 30 18:48:14.539285 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.539252 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg\" (UID: \"5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg" Oct 30 18:48:14.539449 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.539317 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fqqrj\" (UniqueName: \"kubernetes.io/projected/5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19-kube-api-access-fqqrj\") pod \"payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg\" (UID: \"5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg" Oct 30 18:48:14.540760 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.540735 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg\" (UID: \"5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg" Oct 30 18:48:14.558698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.558673 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fqqrj\" (UniqueName: \"kubernetes.io/projected/5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19-kube-api-access-fqqrj\") pod \"payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg\" (UID: \"5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg" Oct 30 18:48:14.563644 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.563616 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf" Oct 30 18:48:14.648515 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.648478 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg" Oct 30 18:48:14.857708 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.857638 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf"] Oct 30 18:48:14.860673 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:48:14.860646 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb309a532_1af6_4ec7_9f02_76cdeb71ae2b.slice/crio-6f3db363ef30e6ca753b5a8919035b6b5deb991f47a0700f87e9296b693b277e WatchSource:0}: Error finding container 6f3db363ef30e6ca753b5a8919035b6b5deb991f47a0700f87e9296b693b277e: Status 404 returned error can't find the container with id 6f3db363ef30e6ca753b5a8919035b6b5deb991f47a0700f87e9296b693b277e Oct 30 18:48:14.960127 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.960104 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg"] Oct 30 18:48:14.962121 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:48:14.962090 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e36ee2e_8f4a_42e7_9ea0_c70668eb2b19.slice/crio-c96a998911403a95ed8aeaa71e4143db49b0119d9305c254289f1edfc581d49b WatchSource:0}: Error finding container c96a998911403a95ed8aeaa71e4143db49b0119d9305c254289f1edfc581d49b: Status 404 returned error can't find the container with id c96a998911403a95ed8aeaa71e4143db49b0119d9305c254289f1edfc581d49b Oct 30 18:48:14.975054 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.975010 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg" event={"ID":"5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19","Type":"ContainerStarted","Data":"c96a998911403a95ed8aeaa71e4143db49b0119d9305c254289f1edfc581d49b"} Oct 30 18:48:14.976809 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.976781 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" event={"ID":"6eaab134-7688-4e9d-8264-361d268cb91d","Type":"ContainerStarted","Data":"09832d6375d3af0ebfac5945f2b5aa05f7f28d282866fde780bf15bb869c1516"} Oct 30 18:48:14.976909 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.976834 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" event={"ID":"6eaab134-7688-4e9d-8264-361d268cb91d","Type":"ContainerStarted","Data":"0265beee6c0fdffff07953067a4628258d0fe72b6f94188b0f7037b6932a4501"} Oct 30 18:48:14.976909 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.976882 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" Oct 30 18:48:14.978596 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.978575 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf" event={"ID":"b309a532-1af6-4ec7-9f02-76cdeb71ae2b","Type":"ContainerStarted","Data":"f9048909f2848c49d39bd93d3258146bfd2a8fe52cebf61bc0095a7447a58344"} Oct 30 18:48:14.978704 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.978619 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf" event={"ID":"b309a532-1af6-4ec7-9f02-76cdeb71ae2b","Type":"ContainerStarted","Data":"6f3db363ef30e6ca753b5a8919035b6b5deb991f47a0700f87e9296b693b277e"} Oct 30 18:48:14.978704 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:14.978639 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf" Oct 30 18:48:15.018579 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:15.018498 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" podStartSLOduration=1.676845284 podStartE2EDuration="2.01848456s" totalImagesPullingTime="341.639276ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:15.017404749 +0000 UTC m=+2794.333825072" watchObservedRunningTime="2026-10-30 18:48:15.01848456 +0000 UTC m=+2794.334904880" Oct 30 18:48:15.055691 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:15.055638 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf" podStartSLOduration=1.055616868 podStartE2EDuration="1.055616868s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:15.054918165 +0000 UTC m=+2794.371338490" watchObservedRunningTime="2026-10-30 18:48:15.055616868 +0000 UTC m=+2794.372037188" Oct 30 18:48:15.983964 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:15.983931 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg" event={"ID":"5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19","Type":"ContainerStarted","Data":"90f5fd92e03627652dfb4509358710290c62637afbb06f80c276f8214838dca1"} Oct 30 18:48:15.984448 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:15.984148 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg" Oct 30 18:48:16.034535 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:16.034473 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg" podStartSLOduration=2.034460584 podStartE2EDuration="2.034460584s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:48:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:48:16.032721376 +0000 UTC m=+2795.349141690" watchObservedRunningTime="2026-10-30 18:48:16.034460584 +0000 UTC m=+2795.350880902" Oct 30 18:48:20.989941 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:20.989861 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" Oct 30 18:48:25.953629 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:25.953596 5232 patch_prober.go:29] interesting pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Oct 30 18:48:25.954195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:25.953663 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" podUID="6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Oct 30 18:48:25.987714 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:25.987684 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf" Oct 30 18:48:26.052073 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:26.052037 5232 generic.go:356] "Generic (PLEG): container finished" podID="6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" containerID="9094ddab6d3f6d483ea318590664af3c019bc230c6fcb22406c07f81cbb91ee2" exitCode=0 Oct 30 18:48:26.052257 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:26.052109 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" event={"ID":"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f","Type":"ContainerDied","Data":"9094ddab6d3f6d483ea318590664af3c019bc230c6fcb22406c07f81cbb91ee2"} Oct 30 18:48:26.052536 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:26.052487 5232 scope.go:122] "RemoveContainer" containerID="9094ddab6d3f6d483ea318590664af3c019bc230c6fcb22406c07f81cbb91ee2" Oct 30 18:48:26.053819 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:26.053781 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:48:26.053946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:26.053865 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:48:26.053946 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:26.053913 5232 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892008Ki","pods":"250"} Oct 30 18:48:26.992046 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:26.992017 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg" Oct 30 18:48:27.062106 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:27.062071 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" event={"ID":"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f","Type":"ContainerStarted","Data":"b1353a85f433daf012728bd179392bc62a0b8de7d4f6f8f21cfebecea74db385"} Oct 30 18:48:27.067841 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:48:27.067810 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:48:27.567152 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:27.567120 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:27.567359 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:27.567304 5232 patch_prober.go:29] interesting pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Oct 30 18:48:27.567448 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:27.567366 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" podUID="6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Oct 30 18:48:28.567988 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:28.567950 5232 patch_prober.go:29] interesting pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Oct 30 18:48:28.568391 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:28.568010 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" podUID="6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Oct 30 18:48:29.567617 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:29.567583 5232 patch_prober.go:29] interesting pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Oct 30 18:48:29.567801 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:29.567636 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" podUID="6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Oct 30 18:48:30.568217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:30.568185 5232 patch_prober.go:29] interesting pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Oct 30 18:48:30.568654 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:30.568240 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" podUID="6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Oct 30 18:48:31.568145 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:31.568109 5232 patch_prober.go:29] interesting pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" start-of-body= Oct 30 18:48:31.568462 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:31.568164 5232 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" podUID="6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.133.1.33:15021/healthz/ready\": dial tcp 10.133.1.33:15021: connect: connection refused" Oct 30 18:48:32.571317 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:32.571282 5232 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:32.571841 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:32.571689 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:32.572474 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:32.572456 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:48:50.035859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:50.035826 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc"] Oct 30 18:48:50.042413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:50.042377 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-nr4nc"] Oct 30 18:48:50.800651 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:50.800619 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dffd8a90-7349-4064-9c07-a64fec195657" path="/var/lib/kubelet/pods/dffd8a90-7349-4064-9c07-a64fec195657/volumes" Oct 30 18:48:56.079529 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:48:56.079485 5232 scope.go:122] "RemoveContainer" containerID="a29b043ed2db931533c4624d00d2d7665761e33a0e60172554a1d9c7bc4c824e" Oct 30 18:48:57.032324 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:48:57.032286 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:49:15.027730 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:49:15.027688 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch"] Oct 30 18:49:15.039668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:49:15.039642 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-z9gch"] Oct 30 18:49:16.801369 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:49:16.801330 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1cd60005-71c6-41cf-9c41-e4dafe67490c" path="/var/lib/kubelet/pods/1cd60005-71c6-41cf-9c41-e4dafe67490c/volumes" Oct 30 18:49:27.034411 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:49:27.034371 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:49:56.111461 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:49:56.111376 5232 scope.go:122] "RemoveContainer" containerID="900e137bd29fc72adc637b37725cfaaab4056c9bf91c835b86d411fbfcd895b7" Oct 30 18:49:57.038127 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:49:57.038067 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:49:58.077914 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:49:58.077880 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg"] Oct 30 18:49:58.117469 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:49:58.117437 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf"] Oct 30 18:50:03.078693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.078642 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg" podUID="5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19" containerName="payload-processing" containerID="cri-o://90f5fd92e03627652dfb4509358710290c62637afbb06f80c276f8214838dca1" gracePeriod=25 Oct 30 18:50:03.118159 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.118115 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf" podUID="b309a532-1af6-4ec7-9f02-76cdeb71ae2b" containerName="payload-pre-processing" containerID="cri-o://f9048909f2848c49d39bd93d3258146bfd2a8fe52cebf61bc0095a7447a58344" gracePeriod=25 Oct 30 18:50:03.371388 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.371364 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg" Oct 30 18:50:03.400883 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.400860 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf" Oct 30 18:50:03.492099 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.492062 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19-kube-api-access-fqqrj\" (UniqueName: \"kubernetes.io/projected/5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19-kube-api-access-fqqrj\") pod \"5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19\" (UID: \"5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19\") " Oct 30 18:50:03.492099 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.492103 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19-plugins-config-volume\") pod \"5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19\" (UID: \"5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19\") " Oct 30 18:50:03.492342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.492142 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b309a532-1af6-4ec7-9f02-76cdeb71ae2b-kube-api-access-zkz7l\" (UniqueName: \"kubernetes.io/projected/b309a532-1af6-4ec7-9f02-76cdeb71ae2b-kube-api-access-zkz7l\") pod \"b309a532-1af6-4ec7-9f02-76cdeb71ae2b\" (UID: \"b309a532-1af6-4ec7-9f02-76cdeb71ae2b\") " Oct 30 18:50:03.492342 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.492224 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b309a532-1af6-4ec7-9f02-76cdeb71ae2b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b309a532-1af6-4ec7-9f02-76cdeb71ae2b-plugins-config-volume\") pod \"b309a532-1af6-4ec7-9f02-76cdeb71ae2b\" (UID: \"b309a532-1af6-4ec7-9f02-76cdeb71ae2b\") " Oct 30 18:50:03.492611 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.492578 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19-plugins-config-volume" pod "5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19" (UID: "5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:03.492693 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.492632 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b309a532-1af6-4ec7-9f02-76cdeb71ae2b-plugins-config-volume" pod "b309a532-1af6-4ec7-9f02-76cdeb71ae2b" (UID: "b309a532-1af6-4ec7-9f02-76cdeb71ae2b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:03.494552 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.494528 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19-kube-api-access-fqqrj" pod "5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19" (UID: "5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19"). InnerVolumeSpecName "kube-api-access-fqqrj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:03.494725 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.494703 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b309a532-1af6-4ec7-9f02-76cdeb71ae2b-kube-api-access-zkz7l" pod "b309a532-1af6-4ec7-9f02-76cdeb71ae2b" (UID: "b309a532-1af6-4ec7-9f02-76cdeb71ae2b"). InnerVolumeSpecName "kube-api-access-zkz7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:03.523560 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.523492 5232 generic.go:356] "Generic (PLEG): container finished" podID="b309a532-1af6-4ec7-9f02-76cdeb71ae2b" containerID="f9048909f2848c49d39bd93d3258146bfd2a8fe52cebf61bc0095a7447a58344" exitCode=0 Oct 30 18:50:03.523746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.523563 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf" Oct 30 18:50:03.523746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.523564 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf" event={"ID":"b309a532-1af6-4ec7-9f02-76cdeb71ae2b","Type":"ContainerDied","Data":"f9048909f2848c49d39bd93d3258146bfd2a8fe52cebf61bc0095a7447a58344"} Oct 30 18:50:03.523746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.523631 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf" event={"ID":"b309a532-1af6-4ec7-9f02-76cdeb71ae2b","Type":"ContainerDied","Data":"6f3db363ef30e6ca753b5a8919035b6b5deb991f47a0700f87e9296b693b277e"} Oct 30 18:50:03.523746 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.523663 5232 scope.go:122] "RemoveContainer" containerID="f9048909f2848c49d39bd93d3258146bfd2a8fe52cebf61bc0095a7447a58344" Oct 30 18:50:03.524986 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.524963 5232 generic.go:356] "Generic (PLEG): container finished" podID="5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19" containerID="90f5fd92e03627652dfb4509358710290c62637afbb06f80c276f8214838dca1" exitCode=0 Oct 30 18:50:03.525093 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.525029 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg" event={"ID":"5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19","Type":"ContainerDied","Data":"90f5fd92e03627652dfb4509358710290c62637afbb06f80c276f8214838dca1"} Oct 30 18:50:03.525093 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.525064 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg" event={"ID":"5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19","Type":"ContainerDied","Data":"c96a998911403a95ed8aeaa71e4143db49b0119d9305c254289f1edfc581d49b"} Oct 30 18:50:03.525185 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.525128 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg" Oct 30 18:50:03.532863 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.532849 5232 scope.go:122] "RemoveContainer" containerID="f9048909f2848c49d39bd93d3258146bfd2a8fe52cebf61bc0095a7447a58344" Oct 30 18:50:03.533174 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:50:03.533150 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9048909f2848c49d39bd93d3258146bfd2a8fe52cebf61bc0095a7447a58344\": container with ID starting with f9048909f2848c49d39bd93d3258146bfd2a8fe52cebf61bc0095a7447a58344 not found: ID does not exist" containerID="f9048909f2848c49d39bd93d3258146bfd2a8fe52cebf61bc0095a7447a58344" Oct 30 18:50:03.533266 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.533185 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9048909f2848c49d39bd93d3258146bfd2a8fe52cebf61bc0095a7447a58344"} err="failed to get container status \"f9048909f2848c49d39bd93d3258146bfd2a8fe52cebf61bc0095a7447a58344\": rpc error: code = NotFound desc = could not find container \"f9048909f2848c49d39bd93d3258146bfd2a8fe52cebf61bc0095a7447a58344\": container with ID starting with f9048909f2848c49d39bd93d3258146bfd2a8fe52cebf61bc0095a7447a58344 not found: ID does not exist" Oct 30 18:50:03.533266 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.533209 5232 scope.go:122] "RemoveContainer" containerID="90f5fd92e03627652dfb4509358710290c62637afbb06f80c276f8214838dca1" Oct 30 18:50:03.542406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.542389 5232 scope.go:122] "RemoveContainer" containerID="90f5fd92e03627652dfb4509358710290c62637afbb06f80c276f8214838dca1" Oct 30 18:50:03.542692 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:50:03.542673 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"90f5fd92e03627652dfb4509358710290c62637afbb06f80c276f8214838dca1\": container with ID starting with 90f5fd92e03627652dfb4509358710290c62637afbb06f80c276f8214838dca1 not found: ID does not exist" containerID="90f5fd92e03627652dfb4509358710290c62637afbb06f80c276f8214838dca1" Oct 30 18:50:03.542738 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.542704 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"90f5fd92e03627652dfb4509358710290c62637afbb06f80c276f8214838dca1"} err="failed to get container status \"90f5fd92e03627652dfb4509358710290c62637afbb06f80c276f8214838dca1\": rpc error: code = NotFound desc = could not find container \"90f5fd92e03627652dfb4509358710290c62637afbb06f80c276f8214838dca1\": container with ID starting with 90f5fd92e03627652dfb4509358710290c62637afbb06f80c276f8214838dca1 not found: ID does not exist" Oct 30 18:50:03.578079 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.578042 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg"] Oct 30 18:50:03.581396 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.581367 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-783444-69d6d6f468-5bltg"] Oct 30 18:50:03.593050 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.593020 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkz7l\" (UniqueName: \"kubernetes.io/projected/b309a532-1af6-4ec7-9f02-76cdeb71ae2b-kube-api-access-zkz7l\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:50:03.593050 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.593047 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b309a532-1af6-4ec7-9f02-76cdeb71ae2b-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:50:03.593217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.593057 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fqqrj\" (UniqueName: \"kubernetes.io/projected/5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19-kube-api-access-fqqrj\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:50:03.593217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.593066 5232 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19-plugins-config-volume\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:50:03.617643 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.617570 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf"] Oct 30 18:50:03.622698 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:03.622661 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-783444-5f746687-4ltkf"] Oct 30 18:50:04.801086 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:04.801052 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19" path="/var/lib/kubelet/pods/5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19/volumes" Oct 30 18:50:04.801489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:04.801410 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b309a532-1af6-4ec7-9f02-76cdeb71ae2b" path="/var/lib/kubelet/pods/b309a532-1af6-4ec7-9f02-76cdeb71ae2b/volumes" Oct 30 18:50:05.882065 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:05.882033 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks"] Oct 30 18:50:05.882957 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:05.882921 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" podUID="6eaab134-7688-4e9d-8264-361d268cb91d" containerName="maas-api" containerID="cri-o://09832d6375d3af0ebfac5945f2b5aa05f7f28d282866fde780bf15bb869c1516" gracePeriod=30 Oct 30 18:50:05.984780 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:05.984738 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" podUID="6eaab134-7688-4e9d-8264-361d268cb91d" containerName="maas-api" probeResult="failure" output="Get \"https://10.133.1.34:8443/health\": dial tcp 10.133.1.34:8443: connect: connection refused" Oct 30 18:50:06.129473 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.129436 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" Oct 30 18:50:06.218234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.218135 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6eaab134-7688-4e9d-8264-361d268cb91d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eaab134-7688-4e9d-8264-361d268cb91d-openshift-service-ca\") pod \"6eaab134-7688-4e9d-8264-361d268cb91d\" (UID: \"6eaab134-7688-4e9d-8264-361d268cb91d\") " Oct 30 18:50:06.218234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.218186 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6eaab134-7688-4e9d-8264-361d268cb91d-kube-api-access-kfmrc\" (UniqueName: \"kubernetes.io/projected/6eaab134-7688-4e9d-8264-361d268cb91d-kube-api-access-kfmrc\") pod \"6eaab134-7688-4e9d-8264-361d268cb91d\" (UID: \"6eaab134-7688-4e9d-8264-361d268cb91d\") " Oct 30 18:50:06.218234 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.218218 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eaab134-7688-4e9d-8264-361d268cb91d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eaab134-7688-4e9d-8264-361d268cb91d-maas-api-tls\") pod \"6eaab134-7688-4e9d-8264-361d268cb91d\" (UID: \"6eaab134-7688-4e9d-8264-361d268cb91d\") " Oct 30 18:50:06.218577 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.218253 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6eaab134-7688-4e9d-8264-361d268cb91d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eaab134-7688-4e9d-8264-361d268cb91d-metrics-certs\") pod \"6eaab134-7688-4e9d-8264-361d268cb91d\" (UID: \"6eaab134-7688-4e9d-8264-361d268cb91d\") " Oct 30 18:50:06.218633 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.218568 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6eaab134-7688-4e9d-8264-361d268cb91d-openshift-service-ca" pod "6eaab134-7688-4e9d-8264-361d268cb91d" (UID: "6eaab134-7688-4e9d-8264-361d268cb91d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:06.218699 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.218680 5232 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6eaab134-7688-4e9d-8264-361d268cb91d-openshift-service-ca\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:50:06.220618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.220595 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6eaab134-7688-4e9d-8264-361d268cb91d-kube-api-access-kfmrc" pod "6eaab134-7688-4e9d-8264-361d268cb91d" (UID: "6eaab134-7688-4e9d-8264-361d268cb91d"). InnerVolumeSpecName "kube-api-access-kfmrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:06.220689 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.220619 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eaab134-7688-4e9d-8264-361d268cb91d-maas-api-tls" pod "6eaab134-7688-4e9d-8264-361d268cb91d" (UID: "6eaab134-7688-4e9d-8264-361d268cb91d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:50:06.220796 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.220783 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6eaab134-7688-4e9d-8264-361d268cb91d-metrics-certs" pod "6eaab134-7688-4e9d-8264-361d268cb91d" (UID: "6eaab134-7688-4e9d-8264-361d268cb91d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 18:50:06.319889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.319853 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kfmrc\" (UniqueName: \"kubernetes.io/projected/6eaab134-7688-4e9d-8264-361d268cb91d-kube-api-access-kfmrc\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:50:06.319889 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.319888 5232 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6eaab134-7688-4e9d-8264-361d268cb91d-maas-api-tls\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:50:06.320102 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.319902 5232 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6eaab134-7688-4e9d-8264-361d268cb91d-metrics-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:50:06.543559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.543486 5232 generic.go:356] "Generic (PLEG): container finished" podID="6eaab134-7688-4e9d-8264-361d268cb91d" containerID="09832d6375d3af0ebfac5945f2b5aa05f7f28d282866fde780bf15bb869c1516" exitCode=0 Oct 30 18:50:06.543559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.543545 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" event={"ID":"6eaab134-7688-4e9d-8264-361d268cb91d","Type":"ContainerDied","Data":"09832d6375d3af0ebfac5945f2b5aa05f7f28d282866fde780bf15bb869c1516"} Oct 30 18:50:06.543810 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.543578 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" Oct 30 18:50:06.543810 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.543606 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks" event={"ID":"6eaab134-7688-4e9d-8264-361d268cb91d","Type":"ContainerDied","Data":"0265beee6c0fdffff07953067a4628258d0fe72b6f94188b0f7037b6932a4501"} Oct 30 18:50:06.543810 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.543629 5232 scope.go:122] "RemoveContainer" containerID="09832d6375d3af0ebfac5945f2b5aa05f7f28d282866fde780bf15bb869c1516" Oct 30 18:50:06.554379 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.554359 5232 scope.go:122] "RemoveContainer" containerID="09832d6375d3af0ebfac5945f2b5aa05f7f28d282866fde780bf15bb869c1516" Oct 30 18:50:06.554740 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:50:06.554718 5232 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09832d6375d3af0ebfac5945f2b5aa05f7f28d282866fde780bf15bb869c1516\": container with ID starting with 09832d6375d3af0ebfac5945f2b5aa05f7f28d282866fde780bf15bb869c1516 not found: ID does not exist" containerID="09832d6375d3af0ebfac5945f2b5aa05f7f28d282866fde780bf15bb869c1516" Oct 30 18:50:06.554822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.554750 5232 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09832d6375d3af0ebfac5945f2b5aa05f7f28d282866fde780bf15bb869c1516"} err="failed to get container status \"09832d6375d3af0ebfac5945f2b5aa05f7f28d282866fde780bf15bb869c1516\": rpc error: code = NotFound desc = could not find container \"09832d6375d3af0ebfac5945f2b5aa05f7f28d282866fde780bf15bb869c1516\": container with ID starting with 09832d6375d3af0ebfac5945f2b5aa05f7f28d282866fde780bf15bb869c1516 not found: ID does not exist" Oct 30 18:50:06.597548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.597483 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks"] Oct 30 18:50:06.608802 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.608772 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-783444-b969dfb9b-fd5ks"] Oct 30 18:50:06.801067 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:06.800983 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6eaab134-7688-4e9d-8264-361d268cb91d" path="/var/lib/kubelet/pods/6eaab134-7688-4e9d-8264-361d268cb91d/volumes" Oct 30 18:50:13.398629 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:13.398592 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph"] Oct 30 18:50:13.399137 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:13.398863 5232 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" podUID="6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" containerName="istio-proxy" containerID="cri-o://b1353a85f433daf012728bd179392bc62a0b8de7d4f6f8f21cfebecea74db385" gracePeriod=30 Oct 30 18:50:17.574209 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:17.574160 5232 patch_prober.go:29] interesting pod/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 18:50:17.574905 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:17.574248 5232 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" podUID="6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 18:50:18.623217 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:18.623160 5232 generic.go:356] "Generic (PLEG): container finished" podID="6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" containerID="b1353a85f433daf012728bd179392bc62a0b8de7d4f6f8f21cfebecea74db385" exitCode=0 Oct 30 18:50:18.624049 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:18.623832 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" event={"ID":"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f","Type":"ContainerDied","Data":"b1353a85f433daf012728bd179392bc62a0b8de7d4f6f8f21cfebecea74db385"} Oct 30 18:50:18.624267 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:18.624236 5232 scope.go:122] "RemoveContainer" containerID="9094ddab6d3f6d483ea318590664af3c019bc230c6fcb22406c07f81cbb91ee2" Oct 30 18:50:19.328610 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.327272 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:50:19.482011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.481049 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-data\") pod \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " Oct 30 18:50:19.482011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.481111 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-podinfo\") pod \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " Oct 30 18:50:19.482011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.481152 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-token\" (UniqueName: \"kubernetes.io/projected/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-token\") pod \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " Oct 30 18:50:19.482011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.481201 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-kube-api-access-sl82r\" (UniqueName: \"kubernetes.io/projected/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-kube-api-access-sl82r\") pod \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " Oct 30 18:50:19.482011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.481236 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-workload-socket\") pod \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " Oct 30 18:50:19.482011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.481268 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istiod-ca-cert\") pod \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " Oct 30 18:50:19.482011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.481306 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-credential-socket\") pod \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " Oct 30 18:50:19.482011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.481362 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-envoy\") pod \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " Oct 30 18:50:19.482011 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.481437 5232 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-workload-certs\") pod \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\" (UID: \"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f\") " Oct 30 18:50:19.488540 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.483200 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-workload-certs" pod "6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" (UID: "6d0a2ef2-20b4-4c2f-ae84-119dec8c235f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:50:19.489038 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.488765 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-workload-socket" pod "6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" (UID: "6d0a2ef2-20b4-4c2f-ae84-119dec8c235f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:50:19.489791 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.489473 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istiod-ca-cert" pod "6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" (UID: "6d0a2ef2-20b4-4c2f-ae84-119dec8c235f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 18:50:19.501926 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.501883 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-credential-socket" pod "6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" (UID: "6d0a2ef2-20b4-4c2f-ae84-119dec8c235f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:50:19.502383 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.502362 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-data" pod "6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" (UID: "6d0a2ef2-20b4-4c2f-ae84-119dec8c235f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:50:19.508089 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.504936 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-envoy" pod "6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" (UID: "6d0a2ef2-20b4-4c2f-ae84-119dec8c235f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 18:50:19.569537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.563699 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-podinfo" pod "6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" (UID: "6d0a2ef2-20b4-4c2f-ae84-119dec8c235f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 30 18:50:19.572022 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.571970 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-token" pod "6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" (UID: "6d0a2ef2-20b4-4c2f-ae84-119dec8c235f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:19.583668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.583333 5232 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-workload-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:50:19.583668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.583365 5232 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istiod-ca-cert\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:50:19.583668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.583379 5232 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-credential-socket\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:50:19.583668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.583394 5232 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-envoy\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:50:19.583668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.583409 5232 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-workload-certs\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:50:19.583668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.583422 5232 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-data\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:50:19.583668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.583434 5232 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-podinfo\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:50:19.583668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.583447 5232 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-istio-token\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:50:19.591333 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.591284 5232 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-kube-api-access-sl82r" pod "6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" (UID: "6d0a2ef2-20b4-4c2f-ae84-119dec8c235f"). InnerVolumeSpecName "kube-api-access-sl82r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 18:50:19.676548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.672638 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" event={"ID":"6d0a2ef2-20b4-4c2f-ae84-119dec8c235f","Type":"ContainerDied","Data":"a52bc5b3de8e5017951c85e04410b4066bcc7d57f3a01ecad95b170b10a6db14"} Oct 30 18:50:19.676548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.672747 5232 scope.go:122] "RemoveContainer" containerID="b1353a85f433daf012728bd179392bc62a0b8de7d4f6f8f21cfebecea74db385" Oct 30 18:50:19.676548 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.672912 5232 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph" Oct 30 18:50:19.688080 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.687881 5232 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sl82r\" (UniqueName: \"kubernetes.io/projected/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f-kube-api-access-sl82r\") on node \"ip-10-0-141-106.ec2.internal\" DevicePath \"\"" Oct 30 18:50:19.772750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.772704 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph"] Oct 30 18:50:19.790534 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:19.779391 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-783444-openshift-default-545d7f66fc-nlbph"] Oct 30 18:50:20.800852 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:20.800816 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" path="/var/lib/kubelet/pods/6d0a2ef2-20b4-4c2f-ae84-119dec8c235f/volumes" Oct 30 18:50:27.144394 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:50:27.144346 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:50:56.131718 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:50:56.131684 5232 scope.go:122] "RemoveContainer" containerID="d923b817d58ace855bb9327ae9f361ab00fab0057936482958135981bc80dcac" Oct 30 18:50:57.032173 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:50:57.032137 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:51:20.998618 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:20.998531 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68c97f9884-jbsf4_2000f04f-227c-415d-bc3a-d270dda9e0c6/manager/0.log" Oct 30 18:51:22.609366 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:22.609327 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fc75ddbcb-q62f8_8c447e9c-f711-4c11-9c6d-09a9c28f65a3/maas-api/0.log" Oct 30 18:51:22.992169 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:22.992143 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-24d8e614-9wq22_e0417a2b-5395-48a7-8a61-b7befdaea670/revoke-keys/0.log" Oct 30 18:51:23.181388 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:23.181361 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-h75jn_5d1b778a-57bc-42c7-8568-f94b7b2eb631/revoke-keys/0.log" Oct 30 18:51:23.361480 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:23.361403 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-ztvlj_98e02b7e-baca-4304-a15e-9db6ae881dee/revoke-keys/0.log" Oct 30 18:51:23.937261 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:23.937233 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-7g2bf_edc2fdde-824f-4bd7-950a-34d07337a7bf/postgres/0.log" Oct 30 18:51:27.026021 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:27.025986 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj"] Oct 30 18:51:27.032980 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:51:27.032948 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:51:27.037699 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:27.037676 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-ztvlj"] Oct 30 18:51:28.114723 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:28.114670 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69dd9bcffb-ddqsh_3571d583-d50a-49ff-bf63-ca8eb1823736/kube-auth-proxy/0.log" Oct 30 18:51:28.800522 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:28.800479 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98e02b7e-baca-4304-a15e-9db6ae881dee" path="/var/lib/kubelet/pods/98e02b7e-baca-4304-a15e-9db6ae881dee/volumes" Oct 30 18:51:29.263999 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:29.263964 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-kbsdt_92403b9c-a622-457e-8b9b-a92f86f93b62/istio-proxy/0.log" Oct 30 18:51:29.636559 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:29.636441 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-pxcpb_59d61286-680e-4bd9-9e35-1127f589f8f7/istio-proxy/0.log" Oct 30 18:51:30.011241 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:30.011202 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-wmxgt_4071cbed-2e82-4a41-97a0-ec2a30675d06/istio-proxy/0.log" Oct 30 18:51:30.189921 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:30.189891 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-zzlvr_b075648b-215b-4d0e-b3dc-41ea5ed46ac2/istio-proxy/0.log" Oct 30 18:51:30.557619 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:30.557585 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9w8hc_9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae/payload-pre-processing/0.log" Oct 30 18:51:32.175256 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:32.175229 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-7k4cr_1b2b3e72-d4ac-4407-97ae-421c5ad091a1/main/0.log" Oct 30 18:51:32.554351 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:32.554322 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2_693cdf78-86cc-4f53-9c17-3926b662aeb4/main/0.log" Oct 30 18:51:44.087267 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:44.087235 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68c97f9884-jbsf4_2000f04f-227c-415d-bc3a-d270dda9e0c6/manager/0.log" Oct 30 18:51:45.471646 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:45.471608 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-5fc75ddbcb-q62f8_8c447e9c-f711-4c11-9c6d-09a9c28f65a3/maas-api/0.log" Oct 30 18:51:45.855621 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:45.855581 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-24d8e614-9wq22_e0417a2b-5395-48a7-8a61-b7befdaea670/revoke-keys/0.log" Oct 30 18:51:46.045884 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:46.045855 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-h75jn_5d1b778a-57bc-42c7-8568-f94b7b2eb631/revoke-keys/0.log" Oct 30 18:51:46.599668 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:46.599627 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-7g2bf_edc2fdde-824f-4bd7-950a-34d07337a7bf/postgres/0.log" Oct 30 18:51:49.023959 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:49.023919 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn"] Oct 30 18:51:49.035989 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:49.035964 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-h75jn"] Oct 30 18:51:49.928913 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:49.928858 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69dd9bcffb-ddqsh_3571d583-d50a-49ff-bf63-ca8eb1823736/kube-auth-proxy/0.log" Oct 30 18:51:50.800970 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:50.800937 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5d1b778a-57bc-42c7-8568-f94b7b2eb631" path="/var/lib/kubelet/pods/5d1b778a-57bc-42c7-8568-f94b7b2eb631/volumes" Oct 30 18:51:51.106553 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:51.106440 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-kbsdt_92403b9c-a622-457e-8b9b-a92f86f93b62/istio-proxy/0.log" Oct 30 18:51:51.492805 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:51.492780 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-pxcpb_59d61286-680e-4bd9-9e35-1127f589f8f7/istio-proxy/0.log" Oct 30 18:51:51.886256 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:51.886164 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-wmxgt_4071cbed-2e82-4a41-97a0-ec2a30675d06/istio-proxy/0.log" Oct 30 18:51:52.084241 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:52.084208 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-zzlvr_b075648b-215b-4d0e-b3dc-41ea5ed46ac2/istio-proxy/0.log" Oct 30 18:51:52.465455 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:52.465424 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9w8hc_9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae/payload-pre-processing/0.log" Oct 30 18:51:53.790207 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:53.790167 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-embedding-simulated-kserve-67bf594cd6-7k4cr_1b2b3e72-d4ac-4407-97ae-421c5ad091a1/main/0.log" Oct 30 18:51:54.163604 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:51:54.163502 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-7658b89khc2_693cdf78-86cc-4f53-9c17-3926b662aeb4/main/0.log" Oct 30 18:51:57.032909 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:51:57.032864 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:52:15.025763 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:52:15.025721 5232 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22"] Oct 30 18:52:15.038173 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:52:15.038138 5232 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-24d8e614-9wq22"] Oct 30 18:52:16.800655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:52:16.800620 5232 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e0417a2b-5395-48a7-8a61-b7befdaea670" path="/var/lib/kubelet/pods/e0417a2b-5395-48a7-8a61-b7befdaea670/volumes" Oct 30 18:52:27.034258 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:52:27.034214 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:52:29.885672 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:52:29.885632 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68c97f9884-jbsf4_2000f04f-227c-415d-bc3a-d270dda9e0c6/manager/0.log" Oct 30 18:52:30.177091 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:52:30.177021 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68c97f9884-jbsf4_2000f04f-227c-415d-bc3a-d270dda9e0c6/manager/0.log" Oct 30 18:52:56.173839 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:52:56.173759 5232 scope.go:122] "RemoveContainer" containerID="d915320c905cedc0664885a4edf5b835d219848c796a4d8ecd7c8d162d333003" Oct 30 18:52:56.182237 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:52:56.182215 5232 scope.go:122] "RemoveContainer" containerID="fa28903c1ca9d8662a8f23f1d5649a58860d6b8248677c35c9e41235f6e5afaa" Oct 30 18:52:57.032547 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:52:57.032496 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:53:16.767521 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:16.767477 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68c97f9884-jbsf4_2000f04f-227c-415d-bc3a-d270dda9e0c6/manager/0.log" Oct 30 18:53:17.056667 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:17.056589 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-68c97f9884-jbsf4_2000f04f-227c-415d-bc3a-d270dda9e0c6/manager/0.log" Oct 30 18:53:18.354413 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.354381 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jtmf9/must-gather-xrb4j"] Oct 30 18:53:18.354849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.354776 5232 cpu_manager.go:414] "containerMap: removing container" podUID="6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" containerName="istio-proxy" Oct 30 18:53:18.354849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.354788 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" containerName="istio-proxy" Oct 30 18:53:18.354849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.354798 5232 cpu_manager.go:414] "containerMap: removing container" podUID="b309a532-1af6-4ec7-9f02-76cdeb71ae2b" containerName="payload-pre-processing" Oct 30 18:53:18.354849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.354803 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b309a532-1af6-4ec7-9f02-76cdeb71ae2b" containerName="payload-pre-processing" Oct 30 18:53:18.354849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.354814 5232 cpu_manager.go:414] "containerMap: removing container" podUID="5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19" containerName="payload-processing" Oct 30 18:53:18.354849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.354822 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19" containerName="payload-processing" Oct 30 18:53:18.354849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.354851 5232 cpu_manager.go:414] "containerMap: removing container" podUID="6eaab134-7688-4e9d-8264-361d268cb91d" containerName="maas-api" Oct 30 18:53:18.355109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.354857 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6eaab134-7688-4e9d-8264-361d268cb91d" containerName="maas-api" Oct 30 18:53:18.355109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.354909 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" containerName="istio-proxy" Oct 30 18:53:18.355109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.354918 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="b309a532-1af6-4ec7-9f02-76cdeb71ae2b" containerName="payload-pre-processing" Oct 30 18:53:18.355109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.354925 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e36ee2e-8f4a-42e7-9ea0-c70668eb2b19" containerName="payload-processing" Oct 30 18:53:18.355109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.354933 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="6eaab134-7688-4e9d-8264-361d268cb91d" containerName="maas-api" Oct 30 18:53:18.355109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.354990 5232 cpu_manager.go:414] "containerMap: removing container" podUID="6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" containerName="istio-proxy" Oct 30 18:53:18.355109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.354996 5232 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" containerName="istio-proxy" Oct 30 18:53:18.355109 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.355067 5232 memory_manager.go:365] "RemoveStaleState removing state" podUID="6d0a2ef2-20b4-4c2f-ae84-119dec8c235f" containerName="istio-proxy" Oct 30 18:53:18.357986 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.357969 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jtmf9/must-gather-xrb4j" Oct 30 18:53:18.375406 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.375381 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jtmf9\"/\"openshift-service-ca.crt\"" Oct 30 18:53:18.386314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.386288 5232 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-jtmf9\"/\"default-dockercfg-6tdcw\"" Oct 30 18:53:18.386314 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.386299 5232 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-jtmf9\"/\"kube-root-ca.crt\"" Oct 30 18:53:18.393367 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.393344 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jtmf9/must-gather-xrb4j"] Oct 30 18:53:18.475287 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.475253 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/17ccca57-bc03-4283-a529-e03a955999cc-must-gather-output\") pod \"must-gather-xrb4j\" (UID: \"17ccca57-bc03-4283-a529-e03a955999cc\") " pod="openshift-must-gather-jtmf9/must-gather-xrb4j" Oct 30 18:53:18.475450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.475298 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-smwzt\" (UniqueName: \"kubernetes.io/projected/17ccca57-bc03-4283-a529-e03a955999cc-kube-api-access-smwzt\") pod \"must-gather-xrb4j\" (UID: \"17ccca57-bc03-4283-a529-e03a955999cc\") " pod="openshift-must-gather-jtmf9/must-gather-xrb4j" Oct 30 18:53:18.576478 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.576447 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/17ccca57-bc03-4283-a529-e03a955999cc-must-gather-output\") pod \"must-gather-xrb4j\" (UID: \"17ccca57-bc03-4283-a529-e03a955999cc\") " pod="openshift-must-gather-jtmf9/must-gather-xrb4j" Oct 30 18:53:18.576655 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.576485 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-smwzt\" (UniqueName: \"kubernetes.io/projected/17ccca57-bc03-4283-a529-e03a955999cc-kube-api-access-smwzt\") pod \"must-gather-xrb4j\" (UID: \"17ccca57-bc03-4283-a529-e03a955999cc\") " pod="openshift-must-gather-jtmf9/must-gather-xrb4j" Oct 30 18:53:18.576849 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.576830 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/17ccca57-bc03-4283-a529-e03a955999cc-must-gather-output\") pod \"must-gather-xrb4j\" (UID: \"17ccca57-bc03-4283-a529-e03a955999cc\") " pod="openshift-must-gather-jtmf9/must-gather-xrb4j" Oct 30 18:53:18.597930 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.597899 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-smwzt\" (UniqueName: \"kubernetes.io/projected/17ccca57-bc03-4283-a529-e03a955999cc-kube-api-access-smwzt\") pod \"must-gather-xrb4j\" (UID: \"17ccca57-bc03-4283-a529-e03a955999cc\") " pod="openshift-must-gather-jtmf9/must-gather-xrb4j" Oct 30 18:53:18.671423 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.671347 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jtmf9/must-gather-xrb4j" Oct 30 18:53:18.972403 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.972315 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jtmf9/must-gather-xrb4j"] Oct 30 18:53:18.978888 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:53:18.978858 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod17ccca57_bc03_4283_a529_e03a955999cc.slice/crio-b288ef39cd15d1e0d67c918c90f8e39c2fb8602a502ec2fa823d8a831dfcd486 WatchSource:0}: Error finding container b288ef39cd15d1e0d67c918c90f8e39c2fb8602a502ec2fa823d8a831dfcd486: Status 404 returned error can't find the container with id b288ef39cd15d1e0d67c918c90f8e39c2fb8602a502ec2fa823d8a831dfcd486 Oct 30 18:53:18.980755 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:18.980737 5232 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 18:53:19.548195 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:19.548127 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jtmf9/must-gather-xrb4j" event={"ID":"17ccca57-bc03-4283-a529-e03a955999cc","Type":"ContainerStarted","Data":"b288ef39cd15d1e0d67c918c90f8e39c2fb8602a502ec2fa823d8a831dfcd486"} Oct 30 18:53:20.555486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:20.555445 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jtmf9/must-gather-xrb4j" event={"ID":"17ccca57-bc03-4283-a529-e03a955999cc","Type":"ContainerStarted","Data":"7b87fe9d9997b477385a12a542adc8d748ebeca227965a90319a588db3a8b6c0"} Oct 30 18:53:20.555486 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:20.555494 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jtmf9/must-gather-xrb4j" event={"ID":"17ccca57-bc03-4283-a529-e03a955999cc","Type":"ContainerStarted","Data":"e94289111b32d19a6795e5ed8a904b2f0e0be7c41e5cf7736e0e3234258d00dc"} Oct 30 18:53:20.596965 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:20.595786 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-jtmf9/must-gather-xrb4j" podStartSLOduration=1.893607828 podStartE2EDuration="2.595766493s" totalImagesPullingTime="702.158665ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 18:53:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:20.593741893 +0000 UTC m=+3099.910162217" watchObservedRunningTime="2026-10-30 18:53:20.595766493 +0000 UTC m=+3099.912186814" Oct 30 18:53:22.666685 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:22.666654 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-brgpf_b0df4b9d-89ce-4bb4-b759-68f1b499b52a/global-pull-secret-syncer/0.log" Oct 30 18:53:23.939537 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:23.937924 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zjlpz_40dd1611-7ec9-411f-920d-e79df7273b90/konnectivity-agent/0.log" Oct 30 18:53:24.185916 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:24.181135 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-106.ec2.internal_1bdd65ea12e9d18ae4c0eda6db4fb016/haproxy/0.log" Oct 30 18:53:27.089534 ip-10-0-141-106 kubenswrapper[5232]: E1030 18:53:27.087128 5232 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-b9kd2" containerName="extractor" Oct 30 18:53:32.347183 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.347144 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54a5c756-d540-4a43-b8a0-aacaf05dc8e8/alertmanager/0.log" Oct 30 18:53:32.453446 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.453410 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54a5c756-d540-4a43-b8a0-aacaf05dc8e8/config-reloader/0.log" Oct 30 18:53:32.555922 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.555888 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54a5c756-d540-4a43-b8a0-aacaf05dc8e8/kube-rbac-proxy-web/0.log" Oct 30 18:53:32.666224 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.666135 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54a5c756-d540-4a43-b8a0-aacaf05dc8e8/kube-rbac-proxy/0.log" Oct 30 18:53:32.744154 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.744117 5232 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx"] Oct 30 18:53:32.749044 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.749015 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:32.770759 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.770732 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx"] Oct 30 18:53:32.778489 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.778465 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54a5c756-d540-4a43-b8a0-aacaf05dc8e8/kube-rbac-proxy-metric/0.log" Oct 30 18:53:32.852531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.850058 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eb0002ca-343c-445d-859c-f45b0b37af06-proc\") pod \"perf-node-gather-daemonset-qh4bx\" (UID: \"eb0002ca-343c-445d-859c-f45b0b37af06\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:32.852531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.850118 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eb0002ca-343c-445d-859c-f45b0b37af06-podres\") pod \"perf-node-gather-daemonset-qh4bx\" (UID: \"eb0002ca-343c-445d-859c-f45b0b37af06\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:32.852531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.850150 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb0002ca-343c-445d-859c-f45b0b37af06-lib-modules\") pod \"perf-node-gather-daemonset-qh4bx\" (UID: \"eb0002ca-343c-445d-859c-f45b0b37af06\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:32.852531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.850210 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb0002ca-343c-445d-859c-f45b0b37af06-sys\") pod \"perf-node-gather-daemonset-qh4bx\" (UID: \"eb0002ca-343c-445d-859c-f45b0b37af06\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:32.852531 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.850243 5232 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vzpx\" (UniqueName: \"kubernetes.io/projected/eb0002ca-343c-445d-859c-f45b0b37af06-kube-api-access-5vzpx\") pod \"perf-node-gather-daemonset-qh4bx\" (UID: \"eb0002ca-343c-445d-859c-f45b0b37af06\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:32.892426 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.892396 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54a5c756-d540-4a43-b8a0-aacaf05dc8e8/prom-label-proxy/0.log" Oct 30 18:53:32.951623 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.951500 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eb0002ca-343c-445d-859c-f45b0b37af06-proc\") pod \"perf-node-gather-daemonset-qh4bx\" (UID: \"eb0002ca-343c-445d-859c-f45b0b37af06\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:32.951623 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.951603 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eb0002ca-343c-445d-859c-f45b0b37af06-podres\") pod \"perf-node-gather-daemonset-qh4bx\" (UID: \"eb0002ca-343c-445d-859c-f45b0b37af06\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:32.951859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.951636 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb0002ca-343c-445d-859c-f45b0b37af06-lib-modules\") pod \"perf-node-gather-daemonset-qh4bx\" (UID: \"eb0002ca-343c-445d-859c-f45b0b37af06\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:32.951859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.951646 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/eb0002ca-343c-445d-859c-f45b0b37af06-proc\") pod \"perf-node-gather-daemonset-qh4bx\" (UID: \"eb0002ca-343c-445d-859c-f45b0b37af06\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:32.951859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.951744 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/eb0002ca-343c-445d-859c-f45b0b37af06-lib-modules\") pod \"perf-node-gather-daemonset-qh4bx\" (UID: \"eb0002ca-343c-445d-859c-f45b0b37af06\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:32.951859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.951748 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/eb0002ca-343c-445d-859c-f45b0b37af06-podres\") pod \"perf-node-gather-daemonset-qh4bx\" (UID: \"eb0002ca-343c-445d-859c-f45b0b37af06\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:32.951859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.951788 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb0002ca-343c-445d-859c-f45b0b37af06-sys\") pod \"perf-node-gather-daemonset-qh4bx\" (UID: \"eb0002ca-343c-445d-859c-f45b0b37af06\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:32.951859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.951839 5232 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vzpx\" (UniqueName: \"kubernetes.io/projected/eb0002ca-343c-445d-859c-f45b0b37af06-kube-api-access-5vzpx\") pod \"perf-node-gather-daemonset-qh4bx\" (UID: \"eb0002ca-343c-445d-859c-f45b0b37af06\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:32.951859 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.951859 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/eb0002ca-343c-445d-859c-f45b0b37af06-sys\") pod \"perf-node-gather-daemonset-qh4bx\" (UID: \"eb0002ca-343c-445d-859c-f45b0b37af06\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:32.972091 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.972058 5232 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vzpx\" (UniqueName: \"kubernetes.io/projected/eb0002ca-343c-445d-859c-f45b0b37af06-kube-api-access-5vzpx\") pod \"perf-node-gather-daemonset-qh4bx\" (UID: \"eb0002ca-343c-445d-859c-f45b0b37af06\") " pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:32.997216 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:32.997189 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_54a5c756-d540-4a43-b8a0-aacaf05dc8e8/init-config-reloader/0.log" Oct 30 18:53:33.063219 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:33.063168 5232 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:33.386236 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:33.386202 5232 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx"] Oct 30 18:53:33.402803 ip-10-0-141-106 kubenswrapper[5232]: W1030 18:53:33.402770 5232 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podeb0002ca_343c_445d_859c_f45b0b37af06.slice/crio-a76f0679902a57f1a8435cc00c50552dc0fdc459670613a52528cc40af94952e WatchSource:0}: Error finding container a76f0679902a57f1a8435cc00c50552dc0fdc459670613a52528cc40af94952e: Status 404 returned error can't find the container with id a76f0679902a57f1a8435cc00c50552dc0fdc459670613a52528cc40af94952e Oct 30 18:53:33.584058 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:33.584025 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-866f94cc5d-rv8z2_abadb041-5d27-42f8-9cb9-35b78ec5f3bf/metrics-server/0.log" Oct 30 18:53:33.682232 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:33.682151 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" event={"ID":"eb0002ca-343c-445d-859c-f45b0b37af06","Type":"ContainerStarted","Data":"3d383c3c0c819a4b8d6324c052b3f1d599d172979c6c5b6c26abfbd7f0c4a402"} Oct 30 18:53:33.682232 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:33.682216 5232 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" event={"ID":"eb0002ca-343c-445d-859c-f45b0b37af06","Type":"ContainerStarted","Data":"a76f0679902a57f1a8435cc00c50552dc0fdc459670613a52528cc40af94952e"} Oct 30 18:53:33.682450 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:33.682289 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:33.722602 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:33.722548 5232 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" podStartSLOduration=1.722534199 podStartE2EDuration="1.722534199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 18:53:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 18:53:33.720866951 +0000 UTC m=+3113.037287296" watchObservedRunningTime="2026-10-30 18:53:33.722534199 +0000 UTC m=+3113.038954531" Oct 30 18:53:33.805656 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:33.805454 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5zplp_17beb356-34fc-4c4a-97a4-72f00c649e7b/node-exporter/0.log" Oct 30 18:53:33.909822 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:33.909785 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5zplp_17beb356-34fc-4c4a-97a4-72f00c649e7b/kube-rbac-proxy/0.log" Oct 30 18:53:34.018394 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:34.018359 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-5zplp_17beb356-34fc-4c4a-97a4-72f00c649e7b/init-textfile/0.log" Oct 30 18:53:36.158549 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:36.158494 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d4f86898f-ng4c7_e8a21a50-4101-4fe4-ba87-ad4aa89f88e7/telemeter-client/0.log" Oct 30 18:53:36.263037 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:36.263006 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d4f86898f-ng4c7_e8a21a50-4101-4fe4-ba87-ad4aa89f88e7/reload/0.log" Oct 30 18:53:36.373808 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:36.373757 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-7d4f86898f-ng4c7_e8a21a50-4101-4fe4-ba87-ad4aa89f88e7/kube-rbac-proxy/0.log" Oct 30 18:53:39.712138 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:39.706096 5232 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-jtmf9/perf-node-gather-daemonset-qh4bx" Oct 30 18:53:44.224791 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:44.224757 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-st6k6_54bdc818-2b49-4f72-a168-6958864242fd/dns/0.log" Oct 30 18:53:44.340763 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:44.340711 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-st6k6_54bdc818-2b49-4f72-a168-6958864242fd/kube-rbac-proxy/0.log" Oct 30 18:53:44.709442 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:44.709404 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-td6vr_8d711c2d-0597-4d83-a884-83e50d98378c/dns-node-resolver/0.log" Oct 30 18:53:45.847167 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:45.847137 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-h6lp9_11311f90-d76a-42b5-b6a2-35c0b62bd222/node-ca/0.log" Oct 30 18:53:46.839181 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:46.839150 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69dd9bcffb-ddqsh_3571d583-d50a-49ff-bf63-ca8eb1823736/kube-auth-proxy/0.log" Oct 30 18:53:47.276266 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:47.276225 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-kbsdt_92403b9c-a622-457e-8b9b-a92f86f93b62/istio-proxy/0.log" Oct 30 18:53:47.403172 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:47.403135 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-pxcpb_59d61286-680e-4bd9-9e35-1127f589f8f7/istio-proxy/0.log" Oct 30 18:53:47.545542 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:47.545433 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-wmxgt_4071cbed-2e82-4a41-97a0-ec2a30675d06/istio-proxy/0.log" Oct 30 18:53:47.617230 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:47.617202 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-zzlvr_b075648b-215b-4d0e-b3dc-41ea5ed46ac2/istio-proxy/0.log" Oct 30 18:53:47.766051 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:47.766023 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9w8hc_9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae/payload-pre-processing/0.log" Oct 30 18:53:49.383448 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:49.383405 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69dd9bcffb-ddqsh_3571d583-d50a-49ff-bf63-ca8eb1823736/kube-auth-proxy/0.log" Oct 30 18:53:50.064728 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:50.063914 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-kbsdt_92403b9c-a622-457e-8b9b-a92f86f93b62/istio-proxy/0.log" Oct 30 18:53:50.280581 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:50.280552 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-pxcpb_59d61286-680e-4bd9-9e35-1127f589f8f7/istio-proxy/0.log" Oct 30 18:53:50.498784 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:50.498760 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-wmxgt_4071cbed-2e82-4a41-97a0-ec2a30675d06/istio-proxy/0.log" Oct 30 18:53:50.605810 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:50.605776 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-zzlvr_b075648b-215b-4d0e-b3dc-41ea5ed46ac2/istio-proxy/0.log" Oct 30 18:53:50.820788 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:50.820698 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9w8hc_9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae/payload-pre-processing/0.log" Oct 30 18:53:52.586477 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:52.586447 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-69dd9bcffb-ddqsh_3571d583-d50a-49ff-bf63-ca8eb1823736/kube-auth-proxy/0.log" Oct 30 18:53:52.986750 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:52.986728 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-kbsdt_92403b9c-a622-457e-8b9b-a92f86f93b62/istio-proxy/0.log" Oct 30 18:53:53.121722 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:53.121692 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-pxcpb_59d61286-680e-4bd9-9e35-1127f589f8f7/istio-proxy/0.log" Oct 30 18:53:53.251682 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:53.251620 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-wmxgt_4071cbed-2e82-4a41-97a0-ec2a30675d06/istio-proxy/0.log" Oct 30 18:53:53.315642 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:53.315612 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-77f556d647-zzlvr_b075648b-215b-4d0e-b3dc-41ea5ed46ac2/istio-proxy/0.log" Oct 30 18:53:53.425060 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:53.425028 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-kqvrz_7bd897e6-be5e-4d00-8faf-b2544c55a3f6/serve-healthcheck-canary/0.log" Oct 30 18:53:53.442684 ip-10-0-141-106 kubenswrapper[5232]: I1030 18:53:53.442656 5232 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-7b98bb678c-9w8hc_9b3c5fd0-ba0e-45b0-a1b9-ac38b9dc8fae/payload-pre-processing/0.log"